# Ascent plugin unable to assign ranks? **URL:** \ **Category:** Errors\ **Tags:** plugins, visualisation\ **Created:** [10 November 2025 22:40 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318 "2025-11-10T22:40:03Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![AlmostSurelyRob](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/almostsurelyrob/32/446_2.png) [@AlmostSurelyRob](https://pyfr.discourse.group/u/AlmostSurelyRob)\ **Post date:** [10 November 2025 22:40 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/1 "2025-11-10T22:40:03Z")
Caveat: I think I am working on a cluster where Slurm appears to be misconfigured, but I would like to open this anyhow and ask for advice. Thanks to @WillT, I have the Ascent plugin working and producing output in a serial run for Taylor Green vortex case. When running in parallel even on a single node: ```auto salloc --time 30:00 --ntasks=4 --gpus-per-task=1 --gpus-per-node=4 --hint=nomultithread mpirun pyfr --progress run --backend cuda mesh.pyfrm conf.ini ``` I get ```shell [LOG_CAT_MCAST] No MCAST components selected [LOG_CAT_ML] Failure in hcoll_mcast_base_select [LOG_CAT_ML] component basesmuma is not available but requested in hierarchy: basesmuma,basesmuma,ucx_p2p:basesmsocket,basesmuma,p2p ... [Error] Ascent::publish file: /gpfs/mcc/HCBEACH03/rxb148/rrs59-rxb148/apps/ascent/ascent-v0.9.5/src/libs/ascent/runtimes/ascent_main_runtime.cpp line: 677 message: Local Domain IDs are not unique on ranks: 0 1 2 3 ``` I believe the relevant part of the conflict is this part: ```ini [backend] precision = single rank-allocator = linear [backend-cuda] device-id = local-rank mpi-type = standard ``` ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [10 November 2025 22:56 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/2 "2025-11-10T22:56:17Z")
Can you add a print statement just before here: > to `print(d_str, rank)`. It isn’t quite clear to me how the domain IDs can clash when each rank uses this as its ID. Regards, Freddie. ---
**Author:** ![AlmostSurelyRob](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/almostsurelyrob/32/446_2.png) [@AlmostSurelyRob](https://pyfr.discourse.group/u/AlmostSurelyRob)\ **Post date:** [12 November 2025 13:50 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/3 "2025-11-12T13:50:06Z")
Thanks. After adding the line here’s the print statement: ```shell domain_1_pri 1 domain_2_pri 2 domain_3_pri 3 domain_0_pri 0 domain_1_tet 1 domain_3_tet 3 domain_2_tet 2 domain_0_tet 0 ``` So it looks like they match, but perhaps the tet vs prism is mixing things up a bit? ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [12 November 2025 14:02 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/4 "2025-11-12T14:02:14Z")
Do you have a single element type case you can test with? It might be that a recent change to Ascent has started to require that domain IDs be unique even on the same rank (although looking through the changelog I can’t see anything obvious here). Regards, Freddie. ---
**Author:** ![AlmostSurelyRob](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/almostsurelyrob/32/446_2.png) [@AlmostSurelyRob](https://pyfr.discourse.group/u/AlmostSurelyRob)\ **Post date:** [12 November 2025 14:44 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/5 "2025-11-12T14:44:54Z")
I’ve just rerun it on Taylor-Green vortex from the test cases and I didn’t get the local rank error, but I got a not-so-helpful backtrace: ```auto Loguru caught a signal: SIGTERM Stack trace: 42 0x401075 _start + 37 41 0x7ffa11a7c640 __libc_start_main + 128 40 0x7ffa11a7c590 /usr/lib64/libc.so.6(+0x29590) [0x7ffa11a7c590] 39 0x7ffa11fad5b9 Py_BytesMain + 41 38 0x7ffa11face29 Py_RunMain + 2457 37 0x7ffa11f88e3b /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x251e3b) [0x7ffa11f88e3b] 36 0x7ffa11f8882f /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x25182f) [0x7ffa11f8882f] 35 0x7ffa11f86933 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x24f933) [0x7ffa11f86933] 34 0x7ffa11f865f7 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x24f5f7) [0x7ffa11f865f7] 33 0x7ffa11f86136 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x24f136) [0x7ffa11f86136] 32 0x7ffa11f2defe PyEval_EvalCode + 158 31 0x7ffa11dc67ad _PyEval_EvalFrameDefault + 15789 30 0x7ffa11e1e130 _PyObject_MakeTpCall + 144 29 0x7ffa11e97248 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x160248) [0x7ffa11e97248] 28 0x7ffa11e9e6f4 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x1676f4) [0x7ffa11e9e6f4] 27 0x7ffa11e1fcd5 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0xe8cd5) [0x7ffa11e1fcd5] 26 0x7ffa11e1fb47 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0xe8b47) [0x7ffa11e1fb47] 25 0x7ffa11dc320e _PyEval_EvalFrameDefault + 2062 24 0x7ffa11e213cf /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0xea3cf) [0x7ffa11e213cf] 23 0x7ffa11dc7b71 _PyEval_EvalFrameDefault + 20849 22 0x7ffa11e1e130 _PyObject_MakeTpCall + 144 21 0x7ffa11e9e5a4 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x1675a4) [0x7ffa11e9e5a4] 20 0x7ffa11e1fd30 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0xe8d30) [0x7ffa11e1fd30] 19 0x7ffa11e1fb47 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0xe8b47) [0x7ffa11e1fb47] 18 0x7ffa11dc67ad _PyEval_EvalFrameDefault + 15789 17 0x7ffa11e1e130 _PyObject_MakeTpCall + 144 16 0x7ffa0748ce2a /gpfs/mcc/apps/python3/3.13/lib/python3.13/lib-dynload/_ctypes.cpython-313-x86_64-linux-gnu.so(+0xfe2a) [0x7ffa0748ce2a] 15 0x7ffa07492746 /gpfs/mcc/apps/python3/3.13/lib/python3.13/lib-dynload/_ctypes.cpython-313-x86_64-linux-gnu.so(+0x15746) [0x7ffa07492746] 14 0x7ffa07475556 /usr/lib64/libffi.so.8(+0x4556) [0x7ffa07475556] 13 0x7ffa074788d6 /usr/lib64/libffi.so.8(+0x78d6) [0x7ffa074788d6] 12 0x7ff7656fe630 ascent::Ascent::execute(conduit::Node const&) + 560 11 0x7ff765733ef4 ascent::AscentRuntime::Execute(conduit::Node const&) + 180 10 0x7ff76571d21e ascent::AscentRuntime::AddPublishedMeshInfo() + 846 9 0x7ff7659b9ace conduit::blueprint::mpi::mesh::generate_index(conduit::Node const&, std::string const&, conduit::Node&, ompi_communicator_t*) + 281 8 0x7ff765a2d46f conduit::relay::mpi::all_gather_using_schema(conduit::Node&, conduit::Node&, ompi_communicator_t*) + 3584 7 0x7ffa10e845d2 PMPI_Allgatherv + 402 6 0x7ffa10f07b4d ompi_coll_tuned_allgatherv_intra_dec_fixed + 189 5 0x7ffa10edc18b ompi_coll_base_allgatherv_intra_neighborexchange + 491 4 0x7ffa10edcaf5 ompi_coll_base_sendrecv_actual + 181 3 0x7ffa10e7245b ompi_request_default_wait + 523 2 0x7ffa1096357d ompi_sync_wait_mt + 221 1 0x7ffa10936934 opal_progress + 52 0 0x7ffa107ed200 ucp_worker_progress + 320 ``` so this may be a separate issue? I don’t know if this is useful, but I compiled Ascent 0.9.5 with the following set of options: ```bash cmake \ -B $ascent_build_dir \ -S $ascent_src_dir \ -DCMAKE_BUILD_TYPE=Release \ -DBUILD_SHARED_LIBS=ON \ -DENABLE_CUDA=OFF \ -DCMAKE_CUDA_ARCHITECTURES="90" \ -DCMAKE_INSTALL_PREFIX=$ascent_install_dir \ -DENABLE_TESTS=OFF \ -DENABLE_MPI=ON \ -DCONDUIT_DIR=$CONDUIT_DIR \ -DVTKM_DIR=$VTKM_DIR \ -DENABLE_FIND_MPI=ON \ -DENABLE_VTKH=ON \ -DENABLE_FORTRAN=OFF \ -DENABLE_OPENMP=OFF \ -DENABLE_PYTHON=OFF ``` ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [12 November 2025 15:00 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/6 "2025-11-12T15:00:31Z")
Can you try with MPI switched out for plain MPICH? You’ll need to recompile both mpi4py and Ascent (and friends). Regards, Freddie. ---
**Author:** ![AlmostSurelyRob](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/almostsurelyrob/32/446_2.png) [@AlmostSurelyRob](https://pyfr.discourse.group/u/AlmostSurelyRob)\ **Post date:** [12 November 2025 15:33 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/7 "2025-11-12T15:33:11Z")
Just a note, I have access to two clusters: OLCF Frontier and STFC Mary Coombs. This backtrace above is from Mary with OpenMPI5, but I can rerun the same on Frontier and compare. I can obviously setup plain MPICH on both too, but it would mean I would need to recompile HDF5 too. ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [12 November 2025 15:38 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/8 "2025-11-12T15:38:08Z")
I’m not sure why you would need to recompile HDF5. Current PyFR does not make use of any of the parallel functionality of HDF5 and so any build will do. For segfaults in this area we only accept the output of MPICH; everything else (vendor variants of MPICH and any builds of OpenMPI) are just far too buggy. Regards, Freddie. ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [12 November 2025 18:31 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/9 "2025-11-12T18:31:28Z")
This > > > This resolves an issue with recent versions of Ascent. should resolve your original issue. It only seems to occur on newer versions of Ascent but it is likely that the code has always been wrong. Regards, Freddie. ---
**Author:** ![AlmostSurelyRob](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/almostsurelyrob/32/446_2.png) [@AlmostSurelyRob](https://pyfr.discourse.group/u/AlmostSurelyRob)\ **Post date:** [27 November 2025 12:12 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/10 "2025-11-27T12:12:12Z")
@fdw , sorry I only came back to this today as I had to focus on generating some results on coarse meshes when Ascent wasn’t critical. I am going to update PyFR and attempt it today and if still failing will do the MPICH as per your previous suggestion. ---
**Author:** ![AlmostSurelyRob](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/almostsurelyrob/32/446_2.png) [@AlmostSurelyRob](https://pyfr.discourse.group/u/AlmostSurelyRob)\ **Post date:** [2 December 2025 14:55 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/11 "2025-12-02T14:55:52Z")
Update from me. [This](https://github.com/PyFR/PyFR/pull/507) has indeed fixed the mixed elements for me in the serial run and I could see the entire cross-section as expected. Thanks for a speedy PR. Reproducing on pure MPICH, compiled with gcc 11 i.e. nothing but the `--prefix=...` in the configure and all but essential Conduit, VTKM and Ascent configures works in parallel too. On top of OpenMPI, which is supplied with this cluster, as well as MPICH on Frontier I still get either an error about local ranks (Frontier) or (Mary Coombs) an inscrutable backtrace involving calls to UCX. I will need to dive in and produce some results for Thursday with what I’ve got but will document what I am seeing with these other installations. What would you suggest though? Should I try to chase the error with OMPI/Frontier MPICH? ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [2 December 2025 16:37 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/12 "2025-12-02T16:37:34Z")
Are you sure that on Frontier you are indeed running in parallel (check by having each rank dump it’s ID). Then, confirm you are indeed using the latest version of PyFR on Frontier and not an older version by mistake. Regards, Freddie. ---
**Author:** ![AlmostSurelyRob](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/almostsurelyrob/32/446_2.png) [@AlmostSurelyRob](https://pyfr.discourse.group/u/AlmostSurelyRob)\ **Post date:** [11 March 2026 11:18 UTC](https://pyfr.discourse.group/t/ascent-plugin-unable-to-assign-ranks/1318/13 "2026-03-11T11:18:23Z")
Revisiting this today I found the source of error on Frontier. I wasn’t in fact using the “GPU-aware” or indeed Slingshot when running on many nodes. I needed to force GPU Transport Layer by pre loading the library. I needed both in my submission script: ```shell export MPICH_GPU_SUPPORT_ENABLED=1 export LD_PRELOAD="${CRAY_MPICH_ROOTDIR}/gtl/lib/libmpi_gtl_hsa.so" ``` and now `hip-aware` and multi-node appear to be working fine on Frontier with Ascent. I guess I assumed that this is done in Lmod modules, but maybe it makes sense.