OpenMP scaling of 2D Couette tutorial

General
1

Hello,

I’m eagerly and gradually learning PyFR. I’ve run the 2D Couette flow tutorial in the examples folder bundled with PyFR 1.8. Now I’m struggling to interpret the behaviour.

  1. The example is setup with order p = 2 and a timestep size of dt = 0.00004. When I increase the order to p = 4, the code spits a NaN error. By trial/error, I’ve found the following timestep sizes are required for solution convergence:

p = 2 dt = 4e-5
p = 3 dt = 3e-5
p = 4 dt = 2e-5
p = 5 dt = 1e-5
p = 6 dt = 5e-6

Question 1: How does the polynomial order affect the time step required for solution stability?

  1. For the above runs, I have plotted the solution residuals below:

residuals.png
residuals.png958×536 42.8 KB

The largest decay rate is with p=1. For p=2 to p=5, the residual drops at nearly identical rates. This was unexpected; I expected the residual to converge at a rate proportional to the polynomial order.

Question 2: How does the polynomial order relate to the residuals produced in the tutorial and their convergence rate?

  1. The wallclock times for the solution runs is not predictable.
    For the above cases, the wallclock time to run for a total of 4 s simulation time is below:

Order Elapsed time

p=1 00:46:11

p=2 00:16:56

p=3 00:12:36

p=4 00:19:19

p=5 00:44:35

p=6 70+ hrs estimated (it took 13 hours to reach 0.6 s then I stopped it)

Granted, the time step sizes were marginally different (as per Q1) so it isn’t a completely consistent comparison. But the solution times are tremendously different and unpredictably so.

Question 3: Why do I see non-monotonic variation in the solution time for different polynomial orders?

Notes about my solution:

  • Backend = OpenMP (8 threads; single cpu).
  • I adjusted p and dt, but left everything else the same in the *.ini file.
  • I used the tutorial-supplied *.msh file.

Thanks!

Joshua

2

Hi Joshua,

/Question 1: How does the polynomial order affect the time step required
for solution stability?/

In a very broad sense dt ~ 1/p^2 and so if you increase the polynomial
order you will need to decrease the time-step accordingly.

/Question 2: How does the polynomial order relate to the residuals
produced in the tutorial and their convergence rate?/

The polynomial order has no substantial impact on the rate of
convergence per-se. However, it does have an impact on the minimum
residual which can be obtained (for a higher polynomial order enables
you to get closer to the true solution).

/Question 3: Why do I see non-monotonic variation in the solution time
for different polynomial orders?/

The Couette flow test case is pathological in the sense it is extremely
small. Run-times are therefore dominated by overhead from Python rather
than actually doing real work. Indeed, you will quite possibly even see
an improvement if you restrict PyFR to a single thread/CPU core.

For a real-world problem the run-time will increase greatly as the
polynomial order is increased. Of course, in the real-world, as you
increase the polynomial order you typically coarsen your mesh accordingly.

Regards, Freddie.

3

Thanks Freddie. This clears up a lot.