Turbulnce plugin hanging. commit bbaaefd

Bugs
1

Good morning everyone, I noticed a potential bug from the turbulence plugin: when running a simulation with the turbulence plugin enabled, for a certain amount of time after the start of the simulation, I get no output (with the writer plugin turned on), not even the progress bar. It is like the solver is “sleeping”. After a while it starts working and it works apparently at the same pace as an equivalent simulation without the turbulence plugin (for example now I’m running a big simulation and it started giving output files after six hours). I explained it also on this previous topic: Turbulence plug-in - #10 by rubens

Here I attach the .ini section

[solver-plugin-turbulence]
avg-rho = 1.0
avg-u = 0.3
avg-mach = 0.3
turbulence-intensity = 0.17 ; we had 0.16 0.17 0.18 and 0.19
turbulence-length-scale = 0.3345
sigma = 0.7 ; default value
centre = (-20, -2.5, -21.875)
y-dim = 55
z-dim = 43.75
rot-axis = (0, 0, 1)
rot-angle = 0.0

PyFR information

  • PyFR version: 1.15.0 (develop branch)
  • OS: SUSE Linux Enterprise Server 15 SP3
  • System description: Working on AMD GPUs
  • Compiler and version: Python 3.10.13
  • Backend (if applicable): hip

Kind regards,
Rubens

2

Hi Rubens,

Thanks for testing this. Super useful to get eyes on it and feedback.

I don’t think its a bug. But there might be room for improvement. Before launching the simulation the plugin goes through and works out which mesh elements are touched by every single injected eddy over the full lifetime of the run. It can then use this information to make the actual injection process more efficient (hence you seeing minimal slowdown when it actually starts running).

Can you give me an idea of the scale of your run (based on the size on the injection plane it seems to be pretty big!). Also, what value is tend set to.

Thanks

Peter

3

Hello Peter, thank you for the quick reply!
The simulation I’m running is pretty big indeed: I’m running it on 80 GPUs and it’s about an airfoil with grid number (evaluated at polynomial order p=4) around 500 mln. The tend is set to 30’000 and I have a timestep dt = 0.00125. I’m asking the solver to output a snapshot every dt-out = 5.

Thank you also for explaining how the plugin behaves!
Kind regards,

Rubens

4

So a serious question here is if you actually expect to run until t=30,000 in a single sitting, as it is the value of tend which matters rather than the output frequency.

Regards, Freddie.

5

Could you try running again but set e.g. tend=10 just to test that this speeds up the startup process.

6

No, my goal is to reach approximately tend=30’000 but in several runnings, as I can get a job running for no more than 24 hours. I will test a simulation with a shorter tend and I will get back to you!
Thank you Freddie, thank you Peter!

Regards,
Rubens

7

I have a smaller run 8mln spt just for testing the plugin. In my case dt=1e-6 and tend=0.6
Using the full inlet plane to generate turbulence it gave me no snapshot and the simulation crashed after 13h.

Any recommendation?

Regards

8

Not sure I follow. What do you mean by ‘8mln spt just for testing the plugin’? Regarding ‘it gave me no snapshot and the simulation crashed after 13h’ how far did it advance in simulation time, and at what frequency was it set to dump solution files?

9

My mesh is 700k hexahedra elements and polynomial order is 4.

Frequency for outputting was set a 0.1
But neither progress bar or snapshot were outputted, I really believe it was stucked at t=0

Despite @rubens I’m using Nvidia A100 cards.

Kind regards

10

What was the error message when it crashed?

11

None just crashed ending the job suddenly.

I will try again and I’ll update you

Regards

12

Hello everyone,
I just made a test simulation with the same mesh but reducing the tend and, as Peter suggested, the time before the solver started to work was reduced.
Thank you for the suggestions!
I will update you if I will notice anything else abput this.
Regards,
Rubens

1 Like