Numpy error when using time average plugin

Hi,

I’m performing an incompressibility(ACM) calculation. I leave a post because there are two problems.

question1.
After saving to 213s, an error occurred and the calculation stopped. (Failed to copy the error)
When I tried to calculate continuously from 213 seconds, the following message occurred. I understand that this is about numpy. Why is it happening and how can it be solved?
There was no problem in calculating compressibility with the same grid (msh) file, and the same error occurred in other incompressibility calculations.

(pyfr14) sk@sk-ubuntu:~/python3.9/pyfr14/airfoil/grid_test/transition_test/Re30000_aoa8/p4_rk2$ mpirun -np 4 pyfr restart -b cuda -p clf5605_c_p4_par4.pyfrm 213.00.pyfrs  incomp_rk.ini
Traceback (most recent call last):
  File "/home/sk/python3.9/pyfr14/bin/pyfr", line 33, in <module>
    sys.exit(load_entry_point('pyfr==1.14.0', 'console_scripts', 'pyfr')())
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/__main__.py", line 115, in main
Traceback (most recent call last):
  File "/home/sk/python3.9/pyfr14/bin/pyfr", line 33, in <module>
    sys.exit(load_entry_point('pyfr==1.14.0', 'console_scripts', 'pyfr')())
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/__main__.py", line 115, in main
    args.process(args)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/__main__.py", line 267, in process_restart
    args.process(args)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/__main__.py", line 267, in process_restart
    _process_common(args, mesh, soln, cfg)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/__main__.py", line 244, in _process_common
    _process_common(args, mesh, soln, cfg)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/__main__.py", line 244, in _process_common
    solver.run()
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/integrators/base.py", line 118, in run
    solver.run()
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/integrators/base.py", line 118, in run
    self.advance_to(t)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/integrators/dual/phys/controllers.py", line 56, in advance_to
    self.advance_to(t)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/integrators/dual/phys/controllers.py", line 56, in advance_to
Traceback (most recent call last):
  File "/home/sk/python3.9/pyfr14/bin/pyfr", line 33, in <module>
    self._accept_step(self.pseudointegrator._idxcurr)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/integrators/dual/phys/controllers.py", line 35, in _accept_step
    sys.exit(load_entry_point('pyfr==1.14.0', 'console_scripts', 'pyfr')())
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/__main__.py", line 115, in main
    csh(self)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 158, in __call__
    args.process(args)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/__main__.py", line 267, in process_restart
    self._accept_step(self.pseudointegrator._idxcurr)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/integrators/dual/phys/controllers.py", line 35, in _accept_step
    _process_common(args, mesh, soln, cfg)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/__main__.py", line 244, in _process_common
    self._init_accumex(intg)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 94, in _init_accumex
    solver.run()
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/integrators/base.py", line 118, in run
    self.prevex = self._eval_acc_exprs(intg)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 136, in _eval_acc_exprs
    csh(self)
    self.advance_to(t)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/integrators/dual/phys/controllers.py", line 56, in advance_to
    return [np.dstack(exs).swapaxes(1, 2) for exs in exprs]
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 136, in <listcomp>
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 158, in __call__
    return [np.dstack(exs).swapaxes(1, 2) for exs in exprs]
  File "<__array_function__ internals>", line 180, in dstack
    self._accept_step(self.pseudointegrator._idxcurr)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/integrators/dual/phys/controllers.py", line 35, in _accept_step
    csh(self)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 158, in __call__
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/numpy/lib/shape_base.py", line 723, in dstack
    self._init_accumex(intg)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 94, in _init_accumex
    self._init_accumex(intg)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 94, in _init_accumex
    self.prevex = self._eval_acc_exprs(intg)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 136, in _eval_acc_exprs
    return [np.dstack(exs).swapaxes(1, 2) for exs in exprs]
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 136, in <listcomp>
    return [np.dstack(exs).swapaxes(1, 2) for exs in exprs]
  File "<__array_function__ internals>", line 180, in dstack
    self.prevex = self._eval_acc_exprs(intg)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 136, in _eval_acc_exprs
Traceback (most recent call last):
  File "/home/sk/python3.9/pyfr14/bin/pyfr", line 33, in <module>
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/numpy/lib/shape_base.py", line 723, in dstack
    return [np.dstack(exs).swapaxes(1, 2) for exs in exprs]
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 136, in <listcomp>
    sys.exit(load_entry_point('pyfr==1.14.0', 'console_scripts', 'pyfr')())
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/__main__.py", line 115, in main
    return _nx.concatenate(arrs, 2)
  File "<__array_function__ internals>", line 180, in concatenate
    return _nx.concatenate(arrs, 2)
  File "<__array_function__ internals>", line 180, in concatenate
    args.process(args)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/__main__.py", line 267, in process_restart
    return [np.dstack(exs).swapaxes(1, 2) for exs in exprs]
  File "<__array_function__ internals>", line 180, in dstack
ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 75 and the array at index 3 has size 1
ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 125 and the array at index 3 has size 1
    _process_common(args, mesh, soln, cfg)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/__main__.py", line 244, in _process_common
    solver.run()
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/integrators/base.py", line 118, in run
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/numpy/lib/shape_base.py", line 723, in dstack
    self.advance_to(t)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/integrators/dual/phys/controllers.py", line 56, in advance_to
    self._accept_step(self.pseudointegrator._idxcurr)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/integrators/dual/phys/controllers.py", line 35, in _accept_step
    csh(self)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 158, in __call__
    self._init_accumex(intg)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 94, in _init_accumex
    self.prevex = self._eval_acc_exprs(intg)
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 136, in _eval_acc_exprs
    return [np.dstack(exs).swapaxes(1, 2) for exs in exprs]
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/pyfr/plugins/tavg.py", line 136, in <listcomp>
    return [np.dstack(exs).swapaxes(1, 2) for exs in exprs]
  File "<__array_function__ internals>", line 180, in dstack
  File "/home/sk/python3.9/pyfr14/lib/python3.9/site-packages/numpy/lib/shape_base.py", line 723, in dstack
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
    return _nx.concatenate(arrs, 2)
  File "<__array_function__ internals>", line 180, in concatenate
ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 75 and the array at index 3 has size 1
    return _nx.concatenate(arrs, 2)
  File "<__array_function__ internals>", line 180, in concatenate
ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 75 and the array at index 3 has size 1
[sk-ubuntu:14667] 3 more processes have sent help message help-mpi-api.txt / mpi-abort
[sk-ubuntu:14667] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

question2.
The error message seems to be related to tavg.
I couldn’t do ‘w’ related tavg for a long time. Therefore, I have removed the variables related to w from the picture below and obtained only the average value for u and v.
tavg

Today, it was confirmed that the average related to w was obtained in the polynomial order = 1, euler analysis.
However, after increasing to polynomial order=4, Runge-Kutta, an error message occurred a long time later and the calculation was terminated. An error message such as question 1 occurred and even restart became impossible.
Why does the error occur when doing the average associated with ‘w’? How can I get a time average related to w?
Does question number one have something to do with it?

Is your case 2d? Becuases if it is then there won’t be a w variable in the system you are solving.

This case was 3d.
I created a mesh using z-direction extrude in pointwise

I see you’re running using Python 3.9. Can you update PyFR, which for the current version will require 3.10, and also update numpy. There were some recent updates to numpy that may be causing you issues.