# Error Running PyFR Across Multiple Nodes **URL:** \ **Category:** Errors\ **Created:** [4 September 2024 21:43 UTC](https://pyfr.discourse.group/t/error-running-pyfr-across-multiple-nodes/1159 "2024-09-04T21:43:16Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![pyfr\_starting\_man](https://avatars.discourse-cdn.com/v4/letter/p/d26b3c/32.png) [@pyfr\_starting\_man](https://pyfr.discourse.group/u/pyfr_starting_man)\ **Post date:** [4 September 2024 21:43 UTC](https://pyfr.discourse.group/t/error-running-pyfr-across-multiple-nodes/1159/1 "2024-09-04T21:43:16Z")
I am running PyFR completely fine when running with 4 GPUs on one node. However, when I am trying to access 20 GPUs across 10 different nodes, I am getting errors. I am using cuda-aware and local-rank in my [backend-cuda]. Is there something else I need to do when running across multiple nodes? Here is the error output: GpuFreq=control\_disabled GpuFreq=control\_disabled GpuFreq=control\_disabled GpuFreq=control\_disabled GpuFreq=control\_disabled GpuFreq=control\_disabled GpuFreq=control\_disabled GpuFreq=control\_disabled GpuFreq=control\_disabled GpuFreq=control\_disabled [p0326.ten.osc.edu:mpi\_rank\_4][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0326.ten.osc.edu:mpi\_rank\_5][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0324.ten.osc.edu:mpi\_rank\_1][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0330.ten.osc.edu:mpi\_rank\_13][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0330.ten.osc.edu:mpi\_rank\_12][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0327.ten.osc.edu:mpi\_rank\_7][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0329.ten.osc.edu:mpi\_rank\_11][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0325.ten.osc.edu:mpi\_rank\_3][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0335.ten.osc.edu:mpi\_rank\_17][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0331.ten.osc.edu:mpi\_rank\_15][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0336.ten.osc.edu:mpi\_rank\_19][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0329.ten.osc.edu:mpi\_rank\_10][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0336.ten.osc.edu:mpi\_rank\_18][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0335.ten.osc.edu:mpi\_rank\_16][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0325.ten.osc.edu:mpi\_rank\_2][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0328.ten.osc.edu:mpi\_rank\_8][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0328.ten.osc.edu:mpi\_rank\_9][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0327.ten.osc.edu:mpi\_rank\_6][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0324.ten.osc.edu:mpi\_rank\_0][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. [p0331.ten.osc.edu:mpi\_rank\_14][rdma\_open\_hca] User specified HCA mlx5\_2 does not have an IP address. Disabling RDMA\_CM based multicast. WARNING: Error in initializing MVAPICH2 ptmalloc library.Continuing without InfiniBand registration cache support. [p0324.ten.osc.edu:mpi\_rank\_0][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0336.ten.osc.edu:mpi\_rank\_19][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0324.ten.osc.edu:mpi\_rank\_1][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0336.ten.osc.edu:mpi\_rank\_18][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0326.ten.osc.edu:mpi\_rank\_5][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0330.ten.osc.edu:mpi\_rank\_13][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0327.ten.osc.edu:mpi\_rank\_7][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0328.ten.osc.edu:mpi\_rank\_8][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0330.ten.osc.edu:mpi\_rank\_12][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0327.ten.osc.edu:mpi\_rank\_6][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0325.ten.osc.edu:mpi\_rank\_2][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0328.ten.osc.edu:mpi\_rank\_9][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0331.ten.osc.edu:mpi\_rank\_14][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0335.ten.osc.edu:mpi\_rank\_16][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0335.ten.osc.edu:mpi\_rank\_17][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0331.ten.osc.edu:mpi\_rank\_15][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0326.ten.osc.edu:mpi\_rank\_4][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0325.ten.osc.edu:mpi\_rank\_3][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0329.ten.osc.edu:mpi\_rank\_11][error\_sighandler] Caught error: Segmentation fault (signal 11) [p0329.ten.osc.edu:mpi\_rank\_10][error\_sighandler] Caught error: Segmentation fault (signal 11) srun: error: p0328: tasks 8-9: Segmentation fault (core dumped) srun: error: p0336: task 18: Segmentation fault (core dumped) srun: error: p0325: task 2: Segmentation fault (core dumped) srun: error: p0335: task 17: Segmentation fault (core dumped) srun: error: p0326: task 4: Segmentation fault (core dumped) srun: error: p0325: task 3: Segmentation fault (core dumped) srun: error: p0335: task 16: Segmentation fault (core dumped) srun: error: p0330: tasks 12-13: Segmentation fault (core dumped) srun: error: p0336: task 19: Segmentation fault (core dumped) srun: error: p0331: tasks 14-15: Segmentation fault (core dumped) srun: error: p0326: task 5: Segmentation fault (core dumped) srun: error: p0327: tasks 6-7: Segmentation fault (core dumped) srun: error: p0324: tasks 0-1: Segmentation fault (core dumped) srun: error: p0329: tasks 10-11: Segmentation fault (core dumped) ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [4 September 2024 21:52 UTC](https://pyfr.discourse.group/t/error-running-pyfr-across-multiple-nodes/1159/2 "2024-09-04T21:52:13Z")
Does it work with `mpi-type = standard`? Regards. Freddie. ---
**Author:** ![pyfr\_starting\_man](https://avatars.discourse-cdn.com/v4/letter/p/d26b3c/32.png) [@pyfr\_starting\_man](https://pyfr.discourse.group/u/pyfr_starting_man)\ **Post date:** [5 September 2024 15:13 UTC](https://pyfr.discourse.group/t/error-running-pyfr-across-multiple-nodes/1159/3 "2024-09-05T15:13:53Z")
No, same exact error. ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [5 September 2024 15:25 UTC](https://pyfr.discourse.group/t/error-running-pyfr-across-multiple-nodes/1159/4 "2024-09-05T15:25:33Z")
Please try with either OpenMPI or MPICH (being sure to recompile mpi4py once you change out your MPI library). Regards, Freddie.