Adapting to Changes in PyFR 1.4.0 Release

General
1

All,

I was attempting to run a case using PyFR—it’s been awhile and a few things have change—but it appears that PyFR doesn’t like my clBLAS library any more under OS X. The pyopencl module seems to have compiled fine when I installed it, and the path to the libclBLAS.dylib file is in my DYLD_LIBRARY_PATH environment. Unfortunately, PyFR seems to be having some issue locating it. Is there another environment variable, or parameter that can be set somewhere to explicitly let PyFR know where it can find this library in my environment? I’ve included the stack trace below:

(venv) [zdavis@Minerva naca0012]$ pyfr run -b opencl -p naca_0012_2d.pyfrm naca_0012_2d.ini

Traceback (most recent call last):
File "/Users/zdavis/Applications/PyFR/pyfr/ctypesutil.py", line 22, in load_library
return ctypes.CDLL(lname)
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ctypes/__init__.py", line 347, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen(libclBLAS.dylib, 6): image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 252, in <module>
main()
File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 109, in main
args.process(args)
File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 230, in process_run
args, NativeReader(args.mesh), None, Inifile.load(args.cfg)
File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 208, in _process_common
backend = get_backend(args.backend, cfg)
File "/Users/zdavis/Applications/PyFR/pyfr/backends/__init__.py", line 12, in get_backend
return subclass_where(BaseBackend, name=name.lower())(cfg)
File "/Users/zdavis/Applications/PyFR/pyfr/backends/opencl/base.py", line 79, in __init__
self._providers = [k(self) for k in kprovs]
File "/Users/zdavis/Applications/PyFR/pyfr/backends/opencl/base.py", line 79, in <listcomp>
self._providers = [k(self) for k in kprovs]
File "/Users/zdavis/Applications/PyFR/pyfr/backends/opencl/clblas.py", line 65, in __init__
self._wrappers = ClBLASWrappers()
File "/Users/zdavis/Applications/PyFR/pyfr/backends/opencl/clblas.py", line 14, in __init__
lib = load_library('clBLAS')
File "/Users/zdavis/Applications/PyFR/pyfr/ctypesutil.py", line 31, in load_library
raise OSError('Unable to load {0}'.format(name))
OSError: Unable to load clBLAS

I’m also getting an invalid character when running using the openmp backend. Again the stack trace is provided below:

(venv) [zdavis@Minerva naca0012]$ mpirun -np 4 pyfr run -b openmp -p naca_0012_2d.pyfrm naca_0012_2d.ini

Traceback (most recent call last):
File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 252, in <module>
main()
File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 109, in main
args.process(args)
File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 230, in process_run
args, NativeReader(args.mesh), None, Inifile.load(args.cfg)
File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 214, in _process_common
solver = get_solver(backend, rallocs, mesh, soln, cfg)
File "/Users/zdavis/Applications/PyFR/pyfr/solvers/__init__.py", line 14, in get_solver
return get_integrator(backend, systemcls, rallocs, mesh, initsoln, cfg)
File "/Users/zdavis/Applications/PyFR/pyfr/integrators/__init__.py", line 43, in get_integrator
return integrator(backend, systemcls, rallocs, mesh, initsoln, cfg)
File "/Users/zdavis/Applications/PyFR/pyfr/integrators/std/controllers.py", line 12, in __init__
super().__init__(*args, **kwargs)
File "/Users/zdavis/Applications/PyFR/pyfr/integrators/std/steppers.py", line 8, in __init__
super().__init__(*args, **kwargs)
File "/Users/zdavis/Applications/PyFR/pyfr/integrators/std/base.py", line 12, in __init__
super().__init__(*args, **kwargs)
File "/Users/zdavis/Applications/PyFR/pyfr/integrators/base.py", line 57, in __init__
self.system = systemcls(backend, rallocs, mesh, initsoln, nreg, cfg)
File "/Users/zdavis/Applications/PyFR/pyfr/solvers/base/system.py", line 54, in __init__
bc_inters = self._load_bc_inters(rallocs, mesh, elemap)
File "/Users/zdavis/Applications/PyFR/pyfr/solvers/base/system.py", line 151, in _load_bc_inters
self.cfg)
File "/Users/zdavis/Applications/PyFR/pyfr/solvers/navstokes/inters.py", line 226, in __init__
self._tpl_c['cpTt'], = self._eval_opts(['cpTt'])
File "/Users/zdavis/Applications/PyFR/pyfr/solvers/baseadvec/inters.py", line 98, in _eval_opts
return [npeval(cfg.getexpr(sect, k), cc) for k in opts]
File "/Users/zdavis/Applications/PyFR/pyfr/solvers/baseadvec/inters.py", line 98, in <listcomp>
return [npeval(cfg.getexpr(sect, k), cc) for k in opts]
File "/Users/zdavis/Applications/PyFR/pyfr/inifile.py", line 79, in getexpr
raise ValueError('Invalid characters in expression')
ValueError: Invalid characters in expression

Is this an issue with my *.ini file? I’ve attached it for reference… Thanks!

Best Regards,

Zach Davis

enc

naca_0012_2d.ini (2.53 KB)

2

Hi Zach,

I was attempting to run a case using PyFR—it’s been awhile and a few
things have change—but it appears that PyFR doesn’t like my clBLAS
library any more under OS X. The pyopencl module seems to have compiled
fine when I installed it, and the path to the libclBLAS.dylib file is in
my DYLD_LIBRARY_PATH environment. Unfortunately, PyFR seems to be
having some issue locating it. Is there another environment variable,
or parameter that can be set somewhere to explicitly let PyFR know where
it can find this library in my environment? I’ve included the stack
trace below:

(venv) [zdavis@Minerva naca0012]$ pyfr run -b opencl -p
naca_0012_2d.pyfrm naca_0012_2d.ini
Traceback (most recent call last):
  File "/Users/zdavis/Applications/PyFR/pyfr/ctypesutil.py", line 22, in
load_library
    return ctypes.CDLL(lname)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ctypes/__init__.py",
line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(libclBLAS.dylib, 6): image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 252, in
<module>
    main()
  File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 109, in
main
    args.process(args)
  File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 230, in
process_run
    args, NativeReader(args.mesh), None, Inifile.load(args.cfg)
  File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 208, in
_process_common
    backend = get_backend(args.backend, cfg)
  File "/Users/zdavis/Applications/PyFR/pyfr/backends/__init__.py", line
12, in get_backend
    return subclass_where(BaseBackend, name=name.lower())(cfg)
  File "/Users/zdavis/Applications/PyFR/pyfr/backends/opencl/base.py",
line 79, in __init__
    self._providers = [k(self) for k in kprovs]
  File "/Users/zdavis/Applications/PyFR/pyfr/backends/opencl/base.py",
line 79, in <listcomp>
    self._providers = [k(self) for k in kprovs]
  File "/Users/zdavis/Applications/PyFR/pyfr/backends/opencl/clblas.py",
line 65, in __init__
    self._wrappers = ClBLASWrappers()
  File "/Users/zdavis/Applications/PyFR/pyfr/backends/opencl/clblas.py",
line 14, in __init__
    lib = load_library('clBLAS')
  File "/Users/zdavis/Applications/PyFR/pyfr/ctypesutil.py", line 31, in
load_library
    raise OSError('Unable to load {0}'.format(name))
OSError: Unable to load clBLAS

The environmental variable PYFR_LIBRARY_PATH can be used to specify
additional search paths for dynamic libraries. Hence,

export PYFR_LIBRARY_PATH=/path/to/dir/with/clBLAS/

should do the trick assuming that there is a library (or a suitable
symlink) to libclBLAS.dylib there. Although this is not 'recommended'
in the sense that it is better to install the library to a location that
is on the search path of the dynamic loader it is 100% supported.

I’m also getting an invalid character when running using the openmp
backend. Again the stack trace is provided below:

(venv) [zdavis@Minerva naca0012]$ mpirun -np 4 pyfr run -b openmp -p
naca_0012_2d.pyfrm naca_0012_2d.ini
Traceback (most recent call last):
  File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 252, in
<module>
    main()
  File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 109, in
main
    args.process(args)
  File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 230, in
process_run
    args, NativeReader(args.mesh), None, Inifile.load(args.cfg)
  File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr", line 214, in
_process_common
    solver = get_solver(backend, rallocs, mesh, soln, cfg)
  File "/Users/zdavis/Applications/PyFR/pyfr/solvers/__init__.py", line
14, in get_solver
    return get_integrator(backend, systemcls, rallocs, mesh, initsoln, cfg)
  File "/Users/zdavis/Applications/PyFR/pyfr/integrators/__init__.py",
line 43, in get_integrator
    return integrator(backend, systemcls, rallocs, mesh, initsoln, cfg)
  File
"/Users/zdavis/Applications/PyFR/pyfr/integrators/std/controllers.py",
line 12, in __init__
    super().__init__(*args, **kwargs)
  File
"/Users/zdavis/Applications/PyFR/pyfr/integrators/std/steppers.py", line
8, in __init__
    super().__init__(*args, **kwargs)
  File "/Users/zdavis/Applications/PyFR/pyfr/integrators/std/base.py",
line 12, in __init__
    super().__init__(*args, **kwargs)
  File "/Users/zdavis/Applications/PyFR/pyfr/integrators/base.py", line
57, in __init__
    self.system = systemcls(backend, rallocs, mesh, initsoln, nreg, cfg)
  File "/Users/zdavis/Applications/PyFR/pyfr/solvers/base/system.py",
line 54, in __init__
    bc_inters = self._load_bc_inters(rallocs, mesh, elemap)
  File "/Users/zdavis/Applications/PyFR/pyfr/solvers/base/system.py",
line 151, in _load_bc_inters
    self.cfg)
  File
"/Users/zdavis/Applications/PyFR/pyfr/solvers/navstokes/inters.py", line
226, in __init__
    self._tpl_c['cpTt'], = self._eval_opts(['cpTt'])
  File
"/Users/zdavis/Applications/PyFR/pyfr/solvers/baseadvec/inters.py", line
98, in _eval_opts
    return [npeval(cfg.getexpr(sect, k), cc) for k in opts]
  File
"/Users/zdavis/Applications/PyFR/pyfr/solvers/baseadvec/inters.py", line
98, in <listcomp>
    return [npeval(cfg.getexpr(sect, k), cc) for k in opts]
  File "/Users/zdavis/Applications/PyFR/pyfr/inifile.py", line 79, in
getexpr
    raise ValueError('Invalid characters in expression')
ValueError: Invalid characters in expression

Is this an issue with my *.ini file? I’ve attached it for reference…
Thanks!

PyFR now allows a variety of expressions, such as those used to define
boundary conditions, to be functions of both time and space. A
consequence of this is that our expression language has to be the lowest
common denominator of what is supported across Python/OpenCL/CUDA/C.
This means that PyFR is no longer able to accept '^' for exponentiation.
Instead, one must use the pow(x, y) function to specify exponents.

For Sutherland's law it looks as if a couple of the constants are
missing. You need to define: cpTref and cpTs in the [constants].

The only other thing is that we have modernised the syntax used in
[solver-plugin-writer] basename. Specifically, the string is now passed
to the format() method hence:

basename = naca_0012_2d-(t:.2f)

should be

basename = naca_0012_2d-{t:.2f}

Hope that helps.

Regards, Freddie.

3

Hi Zach,

My understanding is that the Intel Iris Pro GPUs do not support double
precision. This is something we should probably check for in the OpenCL
backend (there is an API call to determine if double precision compute
is supported or not).

Regards, Freddie.

4

Hi Zach,

Thanks Freddie, I had device-id set to local-rank, and I intended a
value of 1 here. All is well. It appears that throwing the entire
grid as a single partition on my discrete GPU via OpenCL runs much
slower than partitioning the grid into 4 parts and running on the
CPU. Does this make sense to you? Any idea as to why I’m not
getting solution output files?

In general 2D cases are not great from a benchmark standpoint. They
tend to be highly limited by available memory bandwidth and are often
too small to sufficiently load a capable GPU. Moreover, on a good day
it is possible to fit an extremely large portion of the working set into
the last level cache of a modern CPU; as this point it really is no contest.

Hence, it is not unusual for modest CPUs to substantially outperform
high end GPUs in 2D cases.

For file output: rename the section to [soln-plugin-writer] and make
sure that the basename uses {} for variable substitution.

Regards, Freddie.