Ascent plugin unable to assign ranks?

Errors
,
1

Caveat: I think I am working on a cluster where Slurm appears to be misconfigured, but I would like to open this anyhow and ask for advice.

Thanks to @WillT, I have the Ascent plugin working and producing output in a serial run for Taylor Green vortex case. When running in parallel even on a single node:

salloc --time 30:00 --ntasks=4 --gpus-per-task=1 --gpus-per-node=4 --hint=nomultithread
mpirun pyfr --progress run --backend cuda mesh.pyfrm conf.ini

I get

[LOG_CAT_MCAST] No MCAST components selected
[LOG_CAT_ML] Failure in hcoll_mcast_base_select
[LOG_CAT_ML] component basesmuma is not available but requested in hierarchy: basesmuma,basesmuma,ucx_p2p:basesmsocket,basesmuma,p2p
...
[Error] Ascent::publish
file: /gpfs/mcc/HCBEACH03/rxb148/rrs59-rxb148/apps/ascent/ascent-v0.9.5/src/libs/ascent/runtimes/ascent_main_runtime.cpp
line: 677
message:
Local Domain IDs are not unique on ranks: 0 1 2 3

I believe the relevant part of the conflict is this part:

[backend]
precision = single
rank-allocator = linear

[backend-cuda]
device-id = local-rank
mpi-type =  standard
2

Can you add a print statement just before here:

to print(d_str, rank).

It isnā€™t quite clear to me how the domain IDs can clash when each rank uses this as its ID.

Regards, Freddie.

3

Thanks. After adding the line hereā€™s the print statement:

domain_1_pri 1
domain_2_pri 2
domain_3_pri 3
domain_0_pri 0
domain_1_tet 1
domain_3_tet 3
domain_2_tet 2
domain_0_tet 0

So it looks like they match, but perhaps the tet vs prism is mixing things up a bit?

4

Do you have a single element type case you can test with?

It might be that a recent change to Ascent has started to require that domain IDs be unique even on the same rank (although looking through the changelog I canā€™t see anything obvious here).

Regards, Freddie.

5

Iā€™ve just rerun it on Taylor-Green vortex from the test cases and I didnā€™t get the local rank error, but I got a not-so-helpful backtrace:

Loguru caught a signal: SIGTERM
Stack trace:
42            0x401075 _start + 37
41      0x7ffa11a7c640 __libc_start_main + 128
40      0x7ffa11a7c590 /usr/lib64/libc.so.6(+0x29590) [0x7ffa11a7c590]
39      0x7ffa11fad5b9 Py_BytesMain + 41
38      0x7ffa11face29 Py_RunMain + 2457
37      0x7ffa11f88e3b /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x251e3b) [0x7ffa11f88e3b]
36      0x7ffa11f8882f /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x25182f) [0x7ffa11f8882f]
35      0x7ffa11f86933 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x24f933) [0x7ffa11f86933]
34      0x7ffa11f865f7 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x24f5f7) [0x7ffa11f865f7]
33      0x7ffa11f86136 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x24f136) [0x7ffa11f86136]
32      0x7ffa11f2defe PyEval_EvalCode + 158
31      0x7ffa11dc67ad _PyEval_EvalFrameDefault + 15789
30      0x7ffa11e1e130 _PyObject_MakeTpCall + 144
29      0x7ffa11e97248 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x160248) [0x7ffa11e97248]
28      0x7ffa11e9e6f4 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x1676f4) [0x7ffa11e9e6f4]
27      0x7ffa11e1fcd5 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0xe8cd5) [0x7ffa11e1fcd5]
26      0x7ffa11e1fb47 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0xe8b47) [0x7ffa11e1fb47]
25      0x7ffa11dc320e _PyEval_EvalFrameDefault + 2062
24      0x7ffa11e213cf /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0xea3cf) [0x7ffa11e213cf]
23      0x7ffa11dc7b71 _PyEval_EvalFrameDefault + 20849
22      0x7ffa11e1e130 _PyObject_MakeTpCall + 144
21      0x7ffa11e9e5a4 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0x1675a4) [0x7ffa11e9e5a4]
20      0x7ffa11e1fd30 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0xe8d30) [0x7ffa11e1fd30]
19      0x7ffa11e1fb47 /gpfs/mcc/apps/python3/3.13/lib/libpython3.13.so.1.0(+0xe8b47) [0x7ffa11e1fb47]
18      0x7ffa11dc67ad _PyEval_EvalFrameDefault + 15789
17      0x7ffa11e1e130 _PyObject_MakeTpCall + 144
16      0x7ffa0748ce2a /gpfs/mcc/apps/python3/3.13/lib/python3.13/lib-dynload/_ctypes.cpython-313-x86_64-linux-gnu.so(+0xfe2a) [0x7ffa0748ce2a]
15      0x7ffa07492746 /gpfs/mcc/apps/python3/3.13/lib/python3.13/lib-dynload/_ctypes.cpython-313-x86_64-linux-gnu.so(+0x15746) [0x7ffa07492746]
14      0x7ffa07475556 /usr/lib64/libffi.so.8(+0x4556) [0x7ffa07475556]
13      0x7ffa074788d6 /usr/lib64/libffi.so.8(+0x78d6) [0x7ffa074788d6]
12      0x7ff7656fe630 ascent::Ascent::execute(conduit::Node const&) + 560
11      0x7ff765733ef4 ascent::AscentRuntime::Execute(conduit::Node const&) + 180
10      0x7ff76571d21e ascent::AscentRuntime::AddPublishedMeshInfo() + 846
9       0x7ff7659b9ace conduit::blueprint::mpi::mesh::generate_index(conduit::Node const&, std::string const&, conduit::Node&, ompi_communicator_t*) + 281
8       0x7ff765a2d46f conduit::relay::mpi::all_gather_using_schema(conduit::Node&, conduit::Node&, ompi_communicator_t*) + 3584
7       0x7ffa10e845d2 PMPI_Allgatherv + 402
6       0x7ffa10f07b4d ompi_coll_tuned_allgatherv_intra_dec_fixed + 189
5       0x7ffa10edc18b ompi_coll_base_allgatherv_intra_neighborexchange + 491
4       0x7ffa10edcaf5 ompi_coll_base_sendrecv_actual + 181
3       0x7ffa10e7245b ompi_request_default_wait + 523
2       0x7ffa1096357d ompi_sync_wait_mt + 221
1       0x7ffa10936934 opal_progress + 52
0       0x7ffa107ed200 ucp_worker_progress + 320

so this may be a separate issue? I donā€™t know if this is useful, but I compiled Ascent 0.9.5 with the following set of options:

cmake \
    -B $ascent_build_dir \
    -S $ascent_src_dir \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_SHARED_LIBS=ON \
    -DENABLE_CUDA=OFF \
    -DCMAKE_CUDA_ARCHITECTURES="90" \
    -DCMAKE_INSTALL_PREFIX=$ascent_install_dir \
    -DENABLE_TESTS=OFF \
    -DENABLE_MPI=ON \
    -DCONDUIT_DIR=$CONDUIT_DIR \
    -DVTKM_DIR=$VTKM_DIR \
    -DENABLE_FIND_MPI=ON \
    -DENABLE_VTKH=ON \
    -DENABLE_FORTRAN=OFF \
    -DENABLE_OPENMP=OFF \
    -DENABLE_PYTHON=OFF
6

Can you try with MPI switched out for plain MPICH? Youā€™ll need to recompile both mpi4py and Ascent (and friends).

Regards, Freddie.

7

Just a note, I have access to two clusters: OLCF Frontier and STFC Mary Coombs. This backtrace above is from Mary with OpenMPI5, but I can rerun the same on Frontier and compare. I can obviously setup plain MPICH on both too, but it would mean I would need to recompile HDF5 too.

8

Iā€™m not sure why you would need to recompile HDF5. Current PyFR does not make use of any of the parallel functionality of HDF5 and so any build will do.

For segfaults in this area we only accept the output of MPICH; everything else (vendor variants of MPICH and any builds of OpenMPI) are just far too buggy.

Regards, Freddie.

Ascent plugin breaking with alloc_mem_type=cuda
9

This

should resolve your original issue. It only seems to occur on newer versions of Ascent but it is likely that the code has always been wrong.

Regards, Freddie.

1 Like
10

@fdw , sorry I only came back to this today as I had to focus on generating some results on coarse meshes when Ascent wasnā€™t critical. I am going to update PyFR and attempt it today and if still failing will do the MPICH as per your previous suggestion.

11

Update from me.

This has indeed fixed the mixed elements for me in the serial run and I could see the entire cross-section as expected. Thanks for a speedy PR.

Reproducing on pure MPICH, compiled with gcc 11 i.e. nothing but the --prefix=... in the configure and all but essential Conduit, VTKM and Ascent configures works in parallel too. On top of OpenMPI, which is supplied with this cluster, as well as MPICH on Frontier I still get either an error about local ranks (Frontier) or (Mary Coombs) an inscrutable backtrace involving calls to UCX.

I will need to dive in and produce some results for Thursday with what Iā€™ve got but will document what I am seeing with these other installations. What would you suggest though? Should I try to chase the error with OMPI/Frontier MPICH?

12

Are you sure that on Frontier you are indeed running in parallel (check by having each rank dump itā€™s ID). Then, confirm you are indeed using the latest version of PyFR on Frontier and not an older version by mistake.

Regards, Freddie.

13

Revisiting this today I found the source of error on Frontier. I wasnā€™t in fact using the ā€œGPU-awareā€ or indeed Slingshot when running on many nodes. I needed to force GPU Transport Layer by pre loading the library.

I needed both in my submission script:

export MPICH_GPU_SUPPORT_ENABLED=1
export LD_PRELOAD="${CRAY_MPICH_ROOTDIR}/gtl/lib/libmpi_gtl_hsa.so"

and now hip-aware and multi-node appear to be working fine on Frontier with Ascent.

I guess I assumed that this is done in Lmod modules, but maybe it makes sense.