# Poor CPU strong scaling

**URL:** <https://pyfr.discourse.group/t/poor-cpu-strong-scaling/124>\
**Category:** General\
**Created:** [7 December 2016 15:46 UTC](https://pyfr.discourse.group/t/poor-cpu-strong-scaling/124 "2016-12-07T15:46:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Liam\_Doult](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@Liam\_Doult](https://pyfr.discourse.group/u/Liam_Doult)\
**Post date:** [7 December 2016 15:46 UTC](https://pyfr.discourse.group/t/poor-cpu-strong-scaling/124/1 "2016-12-07T15:46:34Z")

</div>

Hi,

I am executing Pyfr 1.5.0 on duel xeon 2630v4 10c per node with 2 nodes. As I scale the mpirun -np up to 40 the performance is drastically reduced.

It also appears to be capped at 50% of the maximum cpu performance.

Any insight is appreciated

Regards  
Liam

---

<div class="post-metadata">

**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\
**Post date:** [7 December 2016 17:41 UTC](https://pyfr.discourse.group/t/poor-cpu-strong-scaling/124/2 "2016-12-07T17:41:15Z")

</div>

Hi Liam,

> I am executing Pyfr 1.5.0 on duel xeon 2630v4 10c per node with 2 nodes.  
> As I scale the mpirun -np up to 40 the performance is drastically reduced.
> 
> It also appears to be capped at 50% of the maximum cpu performance.
> 
> Any insight is appreciated

The performance and scalability of PyFR is heavily determined by the  
type of problem you are running. If the mesh has relatively few  
elements it is not surprising that the performance begins to regress as  
as the number of ranks is increased. Additionally, for some cases PyFR  
can be bound by memory bandwidth and so once you have enough ranks  
inside of a node to saturate the memory bus no improvement should be  
expected from adding additional ranks.

Further, you do not state what compiler/BLAS library you are using and  
what the interconnect is between the nodes. Again, if this is gigabit  
ethernet then poor scalability is not unexpected.

Moreover, PyFR is a hybrid MPI/OpenMP code and so you are almost always  
better off with one MPI rank per NUMA zone.

Regards, Freddie.

---

<div class="post-metadata">

**Author:** ![Liam\_Doult](https://avatars.discourse-cdn.com/v4/letter/l/c4cdca/32.png) [@Liam\_Doult](https://pyfr.discourse.group/u/Liam_Doult)\
**Post date:** [7 December 2016 20:20 UTC](https://pyfr.discourse.group/t/poor-cpu-strong-scaling/124/3 "2016-12-07T20:20:56Z")

</div>

Hi,

So looking deeper into the issue, it appears that running mpirun instantiates 8 processors but only executes on 4.

This is using mpirun -np 8. This occurs on both nodes. so it appears in instantiates 16 instead of 8. This does not happen on any of our other mpirun benchmarks.

We are running centos7, mpi 1.10, and standard gcc for this execution.  
We are executing cube\_hex.

perhaps you can add some insight into why this is happening.

regards  
Liam

---

<div class="post-metadata">

**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\
**Post date:** [7 December 2016 22:13 UTC](https://pyfr.discourse.group/t/poor-cpu-strong-scaling/124/4 "2016-12-07T22:13:08Z")

</div>

Hi Liam,