KeyError on 3D case running with WSL Ubuntu openMP

Errors
,
1

Hi All,

I am running PyFr on Windows WSL Ubuntu with openMP. The 2D example cases work well.

However, when I prepare my own 3D case and tried with the following command: mpiexec -n 2 pyfr run -b openmp -p building_3d.pyfrm building_3d.ini, I am getting a key error.

Would anyone know what cause this error? Is it caused by unsupported environment / improper installation or divergence issue on the simulation? I would appreciate very much if anyone can shed some light on this issue.

Thanks,
Jack

Traceback (most recent call last):
  File "/home/user/ENV3/bin/pyfr", line 8, in <module>
    sys.exit(main())
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/__main__.py", line 117, in main
    args.process(args)
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/__main__.py", line 245, in process_run
    _process_common(
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/__main__.py", line 227, in _process_common
    solver = get_solver(backend, rallocs, mesh, soln, cfg)
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/solvers/__init__.py", line 16, in get_solver
    return get_integrator(backend, systemcls, rallocs, mesh, initsoln, cfg)
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/integrators/__init__.py", line 36, in get_integrator
    return integrator(backend, systemcls, rallocs, mesh, initsoln, cfg)
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/integrators/dual/phys/controllers.py", line 8, in __init__
    super().__init__(*args, **kwargs)
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/integrators/dual/phys/base.py", line 17, in __init__
    self.pseudointegrator = get_pseudo_integrator(
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/integrators/dual/pseudo/__init__.py", line 58, in get_pseudo_integrator
    return DualMultiPIntegrator(backend, systemcls, rallocs, mesh,
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/integrators/dual/pseudo/multip.py", line 123, in __init__
    self.pintgs[l] = lpsint(backend, systemcls, rallocs, mesh,
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/integrators/dual/pseudo/pseudocontrollers.py", line 90, in __init__
    super().__init__(*args, **kwargs)
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/integrators/dual/pseudo/pseudocontrollers.py", line 12, in __init__
    super().__init__(*args, **kwargs)
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/integrators/dual/pseudo/pseudosteppers.py", line 191, in __init__
    super().__init__(*args, **kwargs)
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/integrators/dual/pseudo/base.py", line 50, in __init__
    self.system = systemcls(backend, rallocs, mesh, initsoln,
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/solvers/base/system.py", line 67, in __init__
    self._gen_kernels(eles, int_inters, mpi_inters, bc_inters)
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/solvers/base/system.py", line 188, in _gen_kernels
    kernels[pn, kn].append(kgetter())
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/solvers/baseadvec/elements.py", line 63, in <lambda>
    kernels['tdivtpcorf'] = lambda: self._be.kernel(
  File "/home/user/ENV3/lib/python3.8/site-packages/pyfr/backends/base/backend.py", line 168, in kernel
    raise KeyError(f'Kernel "{name}" has no providers')
KeyError: 'Kernel "mul" has no providers'
2

Hi Jack, wlecome to the forum. Would you be able to post your ini file?

3

Hi Will, below is my ini file: .ini - Google Drive

Let me also post the .msh file here: .msh - Google Drive

[backend]
precision = double
rank-allocator = linear

[constants]
nu = 0.005
Uin = 1.0
Vin = 0.0
Win = 0.0
Pc = 1.0
ac-zeta = 2.5

[solver]
system = ac-navier-stokes
order = 3

[solver-time-integrator]
formulation = dual
scheme = bdf2
pseudo-scheme = rk34
controller = none
pseudo-controller = local-pi
tstart = 0.0
tend = 75.0
dt = 0.0125
pseudo-dt = 0.0025
pseudo-niters-min = 3
pseudo-niters-max = 3
pseudo-resid-norm = l2
pseudo-resid-tol = 1e-12
atol = 1e-6
pseudo-dt-max-mult = 2.5

[solver-dual-time-integrator-multip]
pseudo-dt-fact = 1.75
cycle = [(3, 1), (2, 1), (1, 1), (0, 2), (1, 1), (2, 1), (3, 4)]

[solver-interfaces]
riemann-solver = rusanov
ldg-beta = 0.5
ldg-tau = 0.1

[solver-interfaces-line]
flux-pts = gauss-legendre

[solver-interfaces-quad]
flux-pts = gauss-legendre

[solver-elements-tri]
soln-pts = williams-shunn

[solver-elements-quad]
soln-pts = gauss-legendre

[solver-elements-hex]
soln-pts = gauss-legendre

[soln-plugin-nancheck]
nsteps = 50

[soln-plugin-pseudostats]
flushsteps = 20
file = residual.csv
header = true

[soln-plugin-writer]
dt-out = 5.0
basedir = .
basename = inc_cylinder_2d_{t:.2f}

[soln-bcs-ground]
type = no-slp-wall

[soln-bcs-building]
type = no-slp-wall

[soln-bcs-inlet]
type = ac-char-riem-inv
ac-zeta = 180
p = Pc
u = Uin
v = Vin
w = Win

[soln-bcs-outlet]
type = ac-char-riem-inv
ac-zeta = 180
p = Pc
u = Uin
v = Vin
w = Win

[soln-bcs-top]
type = no-slp-wall

[soln-bcs-right]
type = no-slp-wall

[soln-ics]
u = Uin
v = Vin
w = Win
p = Pc

[backend-openmp]
cc = gcc 
cflags = -L/usr/lib -llapack -lblas

Thanks,
Jack

4

Hi Jack,

Have you tried directly providing the path to libblas in the backend-openmp section? Something like:

cblas= example/path/libblas.so

5

Hi tdzanic, thanks for looking into this. I tried that and I am still getting the same error.

6

Does the same issue come up when you run serially, or is it only when you try to run in parallel?

7

Hi tdzanic,

I just tried running:
pyfr partition 1 building_3d.pyfrm .
pyfr run -b openmp -p building_3d.pyfrm building_3d.ini

But I am still getting the same error.

Jack

8

I am running some diagnostics on the code. It seems that it is trying to find the provider for “mul”.
It could only find OpenMPGiMMiKKernels.mul, but it is giving NotSuitableError: Matrix too dense for GiMMiK. The number of non-zero terms in the matrix is 768 which is larger than the 512 limit.

Should there be another provider for dense matrix multiplication?

Starting: <pyfr.backends.openmp.provider.OpenMPPointwiseKernelProvider object at 0x7fc629cc53a0> mul
Starting: ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_build_arglst', '_build_kernel', '_instantiate_kernel', '_render_kernel', 'backend', 'bccflux', 'bcconu', 'gradcoru', 'gradcorulin', 'intcflux', 'intconu', 'kernel_generator_cls', 'negdivconf', 'register', 'tflux', 'tfluxlin']
Getting: None
Starting: <pyfr.backends.openmp.blasext.OpenMPBlasExtKernels object at 0x7fc629cc5430> mul
Starting: ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_build_kernel', 'axnpby', 'backend', 'copy', 'errest']
Getting: None
Starting: <pyfr.backends.openmp.packing.OpenMPPackingKernels object at 0x7fc629cc5460> mul
Starting: ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_build_kernel', '_sendrecv', 'backend', 'pack', 'recv_pack', 'send_pack', 'unpack']
Getting: None
Starting: <pyfr.backends.openmp.gimmik.OpenMPGiMMiKKernels object at 0x7fc629cc5370> mul
Starting: ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_build_kernel', 'backend', 'max_nnz', 'mul']
Getting: <bound method OpenMPGiMMiKKernels.mul of <pyfr.backends.openmp.gimmik.OpenMPGiMMiKKernels object at 0x7fc629cc5370>>
(<pyfr.backends.openmp.types.OpenMPConstMatrix object at 0x7fc617e49100>, <pyfr.backends.openmp.types.OpenMPMatrixBank object at 0x7fc629cefbb0>) {'out': <pyfr.backends.openmp.types.OpenMPMatrix object at 0x7fc629cef790>}
Starting: <pyfr.backends.openmp.provider.OpenMPPointwiseKernelProvider object at 0x7fc629cc53a0> mul
Starting: ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_build_arglst', '_build_kernel', '_instantiate_kernel', '_render_kernel', 'backend', 'bccflux', 'bcconu', 'gradcoru', 'gradcorulin', 'intcflux', 'intconu', 'kernel_generator_cls', 'negdivconf', 'register', 'tflux', 'tfluxlin']
Getting: None
Starting: <pyfr.backends.openmp.blasext.OpenMPBlasExtKernels object at 0x7fc629cc5430> mul
Starting: ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_build_kernel', 'axnpby', 'backend', 'copy', 'errest']
Getting: None
Starting: <pyfr.backends.openmp.packing.OpenMPPackingKernels object at 0x7fc629cc5460> mul
Starting: ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_build_kernel', '_sendrecv', 'backend', 'pack', 'recv_pack', 'send_pack', 'unpack']
Getting: None
Starting: <pyfr.backends.openmp.gimmik.OpenMPGiMMiKKernels object at 0x7fc629cc5370> mul
Starting: ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_build_kernel', '_memoize_cache', 'backend', 'max_nnz', 'mul']
Getting: <bound method OpenMPGiMMiKKernels.mul of <pyfr.backends.openmp.gimmik.OpenMPGiMMiKKernels object at 0x7fc629cc5370>>
(<pyfr.backends.openmp.types.OpenMPConstMatrix object at 0x7fc608b20b20>, <pyfr.backends.openmp.types.OpenMPMatrix object at 0x7fc629cefc10>) {'out': <pyfr.backends.openmp.types.OpenMPMatrixBank object at 0x7fc629cefc40>}
NotSuitableError: Matrix too dense for GiMMiK
9

At least for pyfr v1.12 you can manually set the cutoff number of non-zeros for gimmik with gimmik-max-nnz = 512 in the openmp backend section. Although this will have an effect on performance. If you have a look at the gimmik paper you can see how the performance changes with matrix size and sparsity.

The libxsmm library can also be an option, see the PyFR OpenMP documentation and the Libxsmm git.

PyFR will also fall back on a blas gemm if the operators are too dense or too big. You may want to check that you have a blas library in your library path. On WSL I think you can do sudo apt install libblas-dev for a quick install, or you can compile your own, or if you have it Intel’s MKL can be a more performant option.

10

There are two options here. The first is to increase gimmik-max-nnz as:

[backend-openmp]
gimmik-max-nnz = 8192

This should work, although performance may be far from optimal. Peak performance can be had through installing libxsmm. Due to bugs you will need to compile the very latest version from GitHub:

Be sure to compile the library with make SHARED=1 NOBLAS=1. Then either add libxsmm.so to your library path or export PYFR_XSMM_LIBRARY_PATH=/path/to/compiled/libxsmm.so

Support for CBLAS (MKL, OpenBLAS, etc) was removed as of v1.12 due to a change in the underlying memory layout.

Regards, Freddie.

11

Thanks! I tried increasing gimmik-max-nnz and it worked.

Jack

12

Hi Freddie,

I met the same problem while running cases with openmp backend. I tried increasing the gimmik-max-nnz number but only works for smaller cases. I also tried to install libxsmm but still not working. Even running example case I have the very same problem without setting gimmik parameters which could work normally before. Any idea about what has happened? Thanks ahead!

Best wishes,
Zhenyang

13

We are gradually migrating the OpenMP backend to requiring libxsmm, and this includes poorly performing fallbacks (cblas, and in the future GiMMiK). Nothing else is competitive in terms of performance.

Regards, Freddie.

14

Hi Freddie,

Thanks for reply. Actually I have compiled the libxsmm library but it is still not working. I have added the shared library to my library path. The same error is still there.

Best wishes,
Zhenyang

15

If you have added the library correctly the following command should run without error:

python -c "from ctypes import CDLL; CDLL('libxsmm.so')"

Regards, Freddie.

16

Hi Freddie,

Yes that command can run without any error. Here is what I got:

zhenyang@blefjell:/scratch/zhenyang/compute/pyfr/channel/channel_local/run$ python -c "from ctypes import CDLL; CDLL('libxsmm.so')"
zhenyang@blefjell:/scratch/zhenyang/compute/pyfr/channel/channel_local/run$ mpirun -np 12 pyfr run -b openmp -p mesh_channel.pyfrm channel_Retau180.ini 
Traceback (most recent call last):
  File "/home/mech/zhenyang/.local/bin/pyfr", line 8, in <module>
    sys.exit(main())
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/__main__.py", line 117, in main
    args.process(args)
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/__main__.py", line 246, in process_run
    args, NativeReader(args.mesh), None, Inifile.load(args.cfg)
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/__main__.py", line 227, in _process_common
    solver = get_solver(backend, rallocs, mesh, soln, cfg)
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/solvers/__init__.py", line 16, in get_solver
    return get_integrator(backend, systemcls, rallocs, mesh, initsoln, cfg)
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/integrators/__init__.py", line 36, in get_integrator
    return integrator(backend, systemcls, rallocs, mesh, initsoln, cfg)
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/integrators/std/controllers.py", line 86, in __init__
    super().__init__(*args, **kwargs)
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/integrators/std/controllers.py", line 14, in __init__
    super().__init__(*args, **kwargs)
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/integrators/std/steppers.py", line 133, in __init__
    super().__init__(*args, **kwargs)
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/integrators/std/base.py", line 28, in __init__
    nregs=self.nregs, cfg=cfg)
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/solvers/base/system.py", line 67, in __init__
    self._gen_kernels(eles, int_inters, mpi_inters, bc_inters)
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/solvers/base/system.py", line 188, in _gen_kernels
    kernels[pn, kn].append(kgetter())
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/solvers/baseadvec/elements.py", line 53, in <lambda>
    out=self._scal_fpts
  File "/home/mech/zhenyang/.local/lib/python3.7/site-packages/pyfr/backends/base/backend.py", line 167, in kernel
    raise KeyError(f'Kernel "{name}" has no providers')
KeyError: 'Kernel "mul" has no providers'
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

Best wishes,
Zhenyang

17

Can you post your config file?

Regards, Freddie.

18

Hi Freddie,

here is ini file

[backend]
precision = double
rank-allocator = linear

;[backend-cuda]
;device-id = local-rank
;mpi-type = standard
;block-1d = 64
;block-2d = 128

[backend-openmp]
cc = gcc
cblas = /usr/lib/x86_64-linux-gnu/atlas/libblas.so. ; Normally I dont need this line; but i tried both are not  working
;gimmik-max-nnz = 8192

[constants]
cInf = 1.0
rhoInf = 1.0
Mach = 0.1
Re = 2767
l = 1.0
gamma = 1.4

uInf = 0.1             ; Mach*cInf
vInf = 0.0
wInf = 0.0
pInf = 0.714285714     ; cInf*cInf*rhoInf/gamma
EInf = 1.7907142849999997          ; pInf/(Gamma-1)+0.5*rhoInf*(uInf*uInf+vInf*vInf)
mu = 0.00003614022406938923    ; uInf*l*rhoInf/Re
Pr = 0.72

[solver]
system = navier-stokes
order = 4

[solver-time-integrator]
scheme = rk45
controller = pi
tstart = 0.0
tend = 100.0

atol = 0.000001
rtol = 0.001
errest-norm = l2
safety-fact = 0.9
min-fact = 0.3
max-fact = 2.5
dt = 0.0001

[solver-interfaces]
riemann-solver = roem
ldg-beta = 0.5
ldg-tau = 0.1

[solver-interfaces-quad]
flux-pts = gauss-legendre
quad-deg = 9
quad-pts = gauss-legendre

[solver-interfaces-line]
flux-pts = gauss-legendre
quad-deg = 9
quad-pts = gauss-legendre

[solver-elements-tri]
soln-pts = williams-shunn
quad-deg = 9

[solver-elements-quad]
soln-pts = gauss-legendre
quad-deg = 9

[solver-elements-hex]
soln-pts = gauss-legendre
quad-deg = 9
quad-pts = gauss-legendre

[soln-plugin-writer]
dt-out = 1
basedir = .
basename = channel_{t:.2f}

[soln-plugin-nancheck]
nsteps = 50

[soln-bcs-wall] ; wall
type = no-slp-adia-wall

[soln-ics]
rho = rhoInf
u = Mach * ( 1.5*(1-y*y) + 0.2*exp(-y*y/0.05)*exp(-x*x/0.05)*exp(-z*z/0.05) )  ; 1.5*M*(1-(2/y/H)**2)
v = 0.2*exp(-y*y/0.05)*exp(-z*z/0.05)*exp(-x*x/0.05)
w = 0.2*exp(-y*y/0.05)*exp(-z*z/0.05)*exp(-x*x/0.05)
p = pInf


[soln-filter]
nsteps = 50     ; check time step
cutoff = 1      ; cut off order
order = 10      ; strength of the filter
alpha = 36.0    ; alpha = -log(ee) where ee is machine precision
19

Can you replace this pass by a print statement (contents do not matter):

and this except pass:

by

except NotSuitableError as e:
    print(e)

Regards, Freddie.

20

Ah, I got the output message

Matrix too large for libxsmm
Matrix too dense for GiMMiK
Traceback (most recent call last):
......
    raise KeyError(f'Kernel "{name}" has no providers')
KeyError: 'Kernel "mul" has no providers'
......

Can that be fixed just increase the cpu number right?

Best wishes,
Zhenyang