# MPI collect wait times informations **URL:** \ **Category:** Cases\ **Tags:** hpc, mpi\ **Created:** [13 June 2023 12:43 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838 "2023-06-13T12:43:26Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Frankx9](https://avatars.discourse-cdn.com/v4/letter/f/3e96dc/32.png) [@Frankx9](https://pyfr.discourse.group/u/Frankx9)\ **Post date:** [13 June 2023 12:43 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/1 "2023-06-13T12:43:26Z")
Hi, I was trying to detect load imbalances but when using collect wait times in the backed section, my solution file only has these informations: ```auto [data] fields = rho,rhou,rhov,rhow,E prefix = soln [backend] rank-allocation = 0,1,2,3,4 [solver-time-integrator] tcurr = 0.02 wall-time = 538.5784029960632 plugin-wall-time-nancheck = 0.03372669219970703 plugin-wall-time-writer = 0.018815994262695312 plugin-wall-time-other = 0.1612997055053711 nsteps = 4096 nacptsteps = 4087 nrjctsteps = 9 nfevals = 20480 ``` There is no information regarding MPI ranks, how I can get those info? Moreover, the key plugin-wall-time-\*\*\*\*\*\* refers to the cumulative time spent out of pure computation (matrix multiplications ecc…) by that plugin? Regards ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [13 June 2023 13:06 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/2 "2023-06-13T13:06:39Z")
This suggests you do not have wait-time collection enabled. Please enable it as: ```auto [backend] collect-wait-times = True ``` Regards, Freddie. ---
**Author:** ![Frankx9](https://avatars.discourse-cdn.com/v4/letter/f/3e96dc/32.png) [@Frankx9](https://pyfr.discourse.group/u/Frankx9)\ **Post date:** [13 June 2023 15:30 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/3 "2023-06-13T15:30:41Z")
Hi Freddie, I have it enabled, may the issue be related to the fact that I’m running on MacOS? Regards ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [13 June 2023 15:49 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/4 "2023-06-13T15:49:11Z")
It should not make a difference. Are you sure you are running a simulation with multiple ranks? Regards, Freddie. ---
**Author:** ![Frankx9](https://avatars.discourse-cdn.com/v4/letter/f/3e96dc/32.png) [@Frankx9](https://pyfr.discourse.group/u/Frankx9)\ **Post date:** [13 June 2023 15:55 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/5 "2023-06-13T15:55:29Z")
Yes I am. From the documentation: `Note that the number of graphs depends on the system, and not all graphs initiate MPI requests. The average amount of time each rank spends waiting for MPI requests per right hand side evaluation can be obtained by vertically summing all of the `-median` fields together.` May be this my case? ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [13 June 2023 16:05 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/6 "2023-06-13T16:05:23Z")
I just tested the code on macOS and got the expected output. Can you dump the configuration file from the solution (`h5dump -d /config ...pyfrs`)? Regards, Freddie. ---
**Author:** ![Frankx9](https://avatars.discourse-cdn.com/v4/letter/f/3e96dc/32.png) [@Frankx9](https://pyfr.discourse.group/u/Frankx9)\ **Post date:** [13 June 2023 16:16 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/7 "2023-06-13T16:16:45Z")
Yup and the output is the one I’ve attached above Regards ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [13 June 2023 16:17 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/8 "2023-06-13T16:17:46Z")
What you provided is the `/stats` record. I would like to see the `/config` record. Regards, Freddie. ---
**Author:** ![Frankx9](https://avatars.discourse-cdn.com/v4/letter/f/3e96dc/32.png) [@Frankx9](https://pyfr.discourse.group/u/Frankx9)\ **Post date:** [13 June 2023 17:00 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/9 "2023-06-13T17:00:25Z")
Got it, reinstalling pyfr solved the issue (don’t know really why tbh) Thanks a lot for the help, one last question how you can interpret those data to identify load imbalances? I have 5 partitions and this is the output: ```auto [backend-wait-times] rhs-graph-0-mean = 0.000893,0.00105,0.00092,0.00193,0.000763 rhs-graph-0-stdev = 0.00112,0.00126,0.00113,0.00162,0.00108 rhs-graph-0-median = 0.000607,0.000825,0.000637,0.00171,0.000433 rhs-graph-1-mean = 0.00281,0.00281,0.00277,0.00456,0.00275 rhs-graph-1-stdev = 0.00279,0.00282,0.0028,0.00349,0.00297 rhs-graph-1-median = 0.00241,0.00239,0.0023,0.00415,0.00199 rhs-graph-2-mean = 0,0,0,0,0 rhs-graph-2-stdev = 0,0,0,0,0 rhs-graph-2-median = 0,0,0,0,0 ``` I can say that given the order of magnitude of the medians of the graphs is roughly the same the partition are actually well balanced? Regards ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [13 June 2023 17:14 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/10 "2023-06-13T17:14:07Z")
The second to last partition (4 of 5) may have too few elements. If you sum the median values it spends more time waiting than other ranks, implying it is finishing its work sooner. Saying that, the difference is not huge and so the balance is likely to still be reasonable. Regards, Freddie. ---
**Author:** ![Frankx9](https://avatars.discourse-cdn.com/v4/letter/f/3e96dc/32.png) [@Frankx9](https://pyfr.discourse.group/u/Frankx9)\ **Post date:** [13 June 2023 18:28 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/11 "2023-06-13T18:28:39Z")
partition 4 has a total wait time of 0.00586 while partition 5 has 0.002423. What would you consider as a huge difference? A order of magnitude of difference? If for example p5 would have a total wait time of 1e-5, how would you repartition those element? ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [13 June 2023 18:58 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/12 "2023-06-13T18:58:46Z")
Assuming the data from your first post holds we do 20480 RHS evaluations. The extra wait time for p4 is `(0.00586 - 0.002423)*20480` which is ~70s. Your wall time is ~538s so this suggests that maybe a 10% improvement in runtime is possible if everything balanced out. Regards, Freddie. ---
**Author:** ![Frankx9](https://avatars.discourse-cdn.com/v4/letter/f/3e96dc/32.png) [@Frankx9](https://pyfr.discourse.group/u/Frankx9)\ **Post date:** [13 June 2023 19:13 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/13 "2023-06-13T19:13:18Z")
Ok thanks a lot, but how can I try to repartition in a more balanced way? my grid only has hex and prism. Is there a particular though process in the weight assignation to metis? Regards ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [13 June 2023 20:06 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/14 "2023-06-13T20:06:54Z")
This is covered in the [performance tuning](https://pyfr.readthedocs.io/en/latest/performance_tuning.html#mixed-grids) section of the documentation. Regards, Freddie. ---
**Author:** ![Frankx9](https://avatars.discourse-cdn.com/v4/letter/f/3e96dc/32.png) [@Frankx9](https://pyfr.discourse.group/u/Frankx9)\ **Post date:** [13 June 2023 23:18 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/15 "2023-06-13T23:18:19Z")
I’m aware of the documentation but to me it is still unclear the precise actions to take. ```auto Element types which are more computationally intensive should be assigned a larger weight than those that are less intensive. … Weights can be specified when partitioning the mesh as `-e shape:weight`. For example, if on a particular system a quadrilateral is found to be 50% more expensive than a triangle this can be specified as: pyfr partition -e quad:3 -e tri:2 ... ``` - Could you explain how the 50% more expensiveness of quad w.r.t to triangles relates to those weights? ```auto If precise profiling data is not available regarding the performance of each element type in a given configuration ``` - How I can get those data per element type to assess how costly they are on my hardware? Regards ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [14 June 2023 00:18 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/16 "2023-06-14T00:18:55Z")
> [@Frankx9](#): > > Could you explain how the 50% more expensiveness of quad w.r.t to triangles relates to those weights? It is encoded by `-e quad:3 -e tri:2` since `3/2 = 1.5` and thus we are giving quads a 50% larger weight than tris. > [@Frankx9](#): > > How I can get those data per element type to assess how costly they are on my hardware? You will need to run a simulation (with the same settings as your actual simulation) on a pure hex mesh and get the timing data, and then again on a pure pri mesh. Then you can obtain a rough cost per element. Regards, Freddie. ---
**Author:** ![Frankx9](https://avatars.discourse-cdn.com/v4/letter/f/3e96dc/32.png) [@Frankx9](https://pyfr.discourse.group/u/Frankx9)\ **Post date:** [14 June 2023 09:12 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/17 "2023-06-14T09:12:12Z")
Ok thanks a lot, but - a priori this cannot be known if only polynomial order is accounted for ? E.g. a ratio between total DoF between each element type - to account properly for the expensiveness of each element type I should also remesh accordingly to have roughly the same DoF given that they differ per element type (?) Regards ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [14 June 2023 11:51 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/18 "2023-06-14T11:51:23Z")
The number of DoF is not an appropriate metric as it does not account for the sparsity present in some operators (hex operators are sparser than prisms, for example), or the differing number of flux points (prisms tend to have more flux points per DoF than hexes). This is why the performance tuning guide suggests benchmarking to find the relative cost and weight accordingly. Regards, Freddie. ---
**Author:** ![Frankx9](https://avatars.discourse-cdn.com/v4/letter/f/3e96dc/32.png) [@Frankx9](https://pyfr.discourse.group/u/Frankx9)\ **Post date:** [14 June 2023 12:34 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/19 "2023-06-14T12:34:29Z")
- once benchmarked, in your experience, the relative weights is basically mostly hardware dependant, case dependant or both? - Lastly, ```auto plugin-wall-time-nancheck = 0.03372669219970703 plugin-wall-time-writer = 0.018815994262695312 plugin-wall-time-other = 0.1612997055053711 nsteps = 4096 ``` those time refer to the total time spent running plugins or do they actually have to be multiplied for the number of steps / nfevals? Regards ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [14 June 2023 13:01 UTC](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838/20 "2023-06-14T13:01:05Z")
The weights are typically hardware and case dependent. The wall clock time given for the plugins is the total amount of time spent in them. Regards, Freddie. [Next page](https://pyfr.discourse.group/t/mpi-collect-wait-times-informations/838.md?page=2)