All,
How do you control whether PyFR runs entirely on CPUs or GPUs? What if you want to run using both? By specifying options within [backend-openmp] that point to a compiler and libraries other than nvcc and cuBlAS or NVBLAS, respectively, are you directing PyFR to run the simulation entirely on CPUs?
Also, in the latest changes merged into master, how do you point to your metis installation? I’ve taken a first attempt to partition a mesh with:
pyfr-mesh partition -p metis 4 naca_0012_2d.pyfrm .
only to get the following stack trace:
Traceback (most recent call last):
File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr-mesh", line 117, in <module>
main()
File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr-mesh", line 113, in main
args.process(args)
File "/Users/zdavis/Applications/PyFR/pyfr/scripts/pyfr-mesh", line 47, in process_partition
part = get_partitioner_by_name(args.partitioner, pwts, opts)
File "/Users/zdavis/Applications/PyFR/pyfr/partitioners/__init__.py", line 10, in get_partitioner_by_name
return subclass_where(BasePartitioner, name=name)(*args, **kwargs)
File "/Users/zdavis/Applications/PyFR/pyfr/partitioners/metis.py", line 139, in __init__
self._wrappers = METISWrappers()
File "/Users/zdavis/Applications/PyFR/pyfr/partitioners/metis.py", line 40, in __init__
lib = load_library('metis')
File "/Users/zdavis/Applications/PyFR/pyfr/ctypesutil.py", line 23, in load_library
raise OSError('Unable to load {0}'.format(name))
OSError: Unable to load metis
Best Regards,
Zach Davis
Hi Zach,
How do you control whether PyFR runs entirely on CPUs or GPUs?
What if you want to run using both? By specifying options within
[backend-openmp] that point to a compiler and libraries other than
nvcc and cuBlAS or NVBLAS, respectively, are you directing PyFR to
run the simulation entirely on CPUs?
This is controlled by the '-b' (backend) switch to pyfr-sim. Passing
'-bcuda' will instruct PyFR to use the CUDA backend. If, instead, one
passes '-bopenmp' then PyFR will use the C/OpenMP backend which
targets CPUs. The [backend-openmp] section of the configuration file
only affects the workings of the C/OpenMP backend.
Heterogeneous runs are a little bit trickier. Firstly, you will want
to be using either MVAPICH2 or OpenMPI. Second, you need to suitably
partition your mesh such that the partition sizes are commensurate
with the relative speeds of the platforms you are running on.
Thirdly, you need to get mpirun to execute a script which looks at the
MPI rank ($OMPI_COMM_WORLD_RANK for OpenMPI) and uses this to decide
if to execute:
pyfr run -b cuda ...
or
pyfr run -b openmp ...
It works, but I'll agree it is complicated. The idea of
simultaneously using CPUs and GPUs to solve the same problem (as
opposed to different problems -- which is what most heterogeneous
codes currently do) is still quite young. As such the support
infrastructure really isn't there (MPI libraries doing 'the wrong
thing'; pinning is especially painful when undertaking such simulations).
Also, in the latest changes merged into master, how do you point
to your metis installation? I’ve taken a first attempt to partition
a mesh with:
pyfr-mesh partition -p metis 4 naca_0012_2d.pyfrm .
This appears to imply that metis (version 5 built as a shared library)
can not be found by the dynamic linker. Find the libmetis.so file and
add the directory to LD_LIBRARY_PATH.
Regards, Freddie.
I had the same problem but did fix it. I found the easiest solution was to use synaptic package manager and install the libmetis-dev. This will install everything you need and set the path.