Dear All,
I’m trying to run PyFR on Xeon Phi 7250 (Knight Landing). There is only one Xeon Phi processor, and no CPU on my computing node. Therefore, I log in to the Xeon Phi from front-end node via ssh, and execute command on Xeon Phi directly.
In this case, how I can write “backend-mic” section in .ini file? Could you tell me sample description of “device-id” and “mkl-root” parameters?
I could run PyFR using “backend-openmp” settings on Xeon Phi. However, I think simulation speed will be faster if I can run PyFR with mic native functions.
Best regards,
Tom
Hi Tom,
I'm trying to run PyFR on Xeon Phi 7250 (Knight Landing). There is only
one Xeon Phi processor, and no CPU on my computing node. Therefore, I
log in to the Xeon Phi from front-end node via ssh, and execute command
on Xeon Phi directly.
In this case, how I can write "backend-mic" section in .ini file? Could
you tell me sample description of "device-id" and "mkl-root" parameters?
I could run PyFR using "backend-openmp" settings on Xeon Phi. However, I
think simulation speed will be faster if I can run PyFR with mic native
functions.
For KNL you want to be using the OpenMP backend. The key here is to
make sure that you get the number of OpenMP threads correct, have MKL
available, use the ICC compiler, and have libxsmm installed. Together,
these should result in rather good performance for non-pathological test
cases.
The MIC backend is for the legacy Knights Corner.
Regards, Freddie.
Hi Freddie,
Thank you for your reply.
I have installed libxsmm according to commands as described below.
make STATIC=0 BLAS=0
make PREFIX= STATIC=0 BLAS=0 install
The make process finished without error. Then, I set parameter in “backend-openmp” section in ini file.
[backend-openmp]
cc = icc
cblas = /lib/libxsmm.so
Are the setting parameters correct? Xeon Phi 7250 have 68 physical cores, and each core can run 4 threads. Therefore, I set $OMP_NUM_THREADS=272.
Could you tell me meanings of “gimmik-max-nnz”, “libxsmm-block-sz” and “libxsmm-max-sz” in “backend-openmp” section in ini file?
Best regards,
Tom
Hi Tom,
Thank you for your reply.
I have installed libxsmm according to commands as described below.
make STATIC=0 BLAS=0
make PREFIX=<libxsmm install path> STATIC=0 BLAS=0 install
The make process finished without error. Then, I set parameter in
"backend-openmp" section in ini file.
[backend-openmp]
cc = icc
cblas = <libxsmm install path>/lib/libxsmm.so
So libxsmm is not a cblas library (that is MKL). All you need to do is
ensure that libxsmm is somewhere where PyFR can find it. If need be you
can point PyFR to the library by doing
export PYFR_XSMM_LIBRARY_PATH=/path/here/to/lib.so
Are the setting parameters correct? Xeon Phi 7250 have 68 physical
cores, and each core can run 4 threads. Therefore, I set
$OMP_NUM_THREADS=272.
For best performance you want one thread per core.
Could you tell me meanings of "gimmik-max-nnz", "libxsmm-block-sz" and
"libxsmm-max-sz" in "backend-openmp" section in ini file?
The first parameter decides the cut-off point at which we will no longer
use the GiMMiK matrix multiplication library. The second parameter
controls the block size for libxsmm, although performance does not
appear to be too sensitive to its value. The final parameter decides
the cut-off point at which we will no longer use libxsmm for matrix
multiplications. The hierarchy is:
1. libxsmm (if available and size < libxsmm-max-sz)
2. GiMMiK (else if nonzeros < gimmik-max-nnz)
3. cblas (else)
The defaults in PyFR are not too bad, but you may be able to get 20-30%
out for certain test cases by playing around with these parameters. The
key is to ensure that libxsmm is available, this is especially important
at lower polynomial orders.
Regards, Freddie.
Hi Freddie,
Thank you for the detail information.
I set the environment variable “PYFR_XSMM_LIBRARY_PATH” in .bashrc
export PYFR_XSMM_LIBRARY_PATH=$HOME/FR/libxmm/lib
In libxmm/lib, there are dynamic libraries like libxsmm*.so.
Then, I set “[backend-openmp]” section in .ini file as described below.
[backend-openmp]
cc = icc
cblas = /opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64_lin_mic/libmkl_rt.so
I got an error “KeyError: “‘mul’ has no providers”” when I execute PyFR. The error could be related to wrong cblas library pass. In mkl lib directory, there are many libraries. What dynamic library I should set in cblas section?
libmkl_ao_worker.so libmkl_cdft_core.so libmkl_intel_thread.so libmkl_sequential.a
libmkl_blacs_intelmpi_ilp64.a libmkl_core.a libmkl_lapack95_ilp64.a libmkl_sequential.so
libmkl_blacs_intelmpi_ilp64.so libmkl_core.so libmkl_lapack95_lp64.a libmkl_tbb_thread.a
libmkl_blacs_intelmpi_lp64.a libmkl_intel_ilp64.a libmkl_rt.so libmkl_tbb_thread.so
libmkl_blacs_intelmpi_lp64.so libmkl_intel_ilp64.so libmkl_scalapack_ilp64.a locale
libmkl_blas95_ilp64.a libmkl_intel_lp64.a libmkl_scalapack_ilp64.so
libmkl_blas95_lp64.a libmkl_intel_lp64.so libmkl_scalapack_lp64.a
libmkl_cdft_core.a libmkl_intel_thread.a libmkl_scalapack_lp64.so
Best regards,
Tom
Hi Tom,
Thank you for the detail information.
I set the environment variable "PYFR_XSMM_LIBRARY_PATH" in .bashrc
The path needs to be the absolute path to the library itself as opposed
to the directory in which it is located.
export PYFR_XSMM_LIBRARY_PATH=$HOME/FR/libxmm/lib
In libxmm/lib, there are dynamic libraries like libxsmm*.so.
Then, I set "[backend-openmp]" section in .ini file as described below.
[backend-openmp]
cc = icc
cblas =
/opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64_lin_mic/libmkl_rt.so
I got an error "KeyError: "'mul' has no providers"" when I execute PyFR.
The error could be related to wrong cblas library pass. In mkl lib
directory, there are many libraries. What dynamic library I should set
in cblas section?
libmkl_ao_worker.so libmkl_cdft_core.so
libmkl_intel_thread.so libmkl_sequential.a
libmkl_blacs_intelmpi_ilp64.a libmkl_core.a
libmkl_lapack95_ilp64.a libmkl_sequential.so
libmkl_blacs_intelmpi_ilp64.so libmkl_core.so
libmkl_lapack95_lp64.a libmkl_tbb_thread.a
libmkl_blacs_intelmpi_lp64.a libmkl_intel_ilp64.a
libmkl_rt.so libmkl_tbb_thread.so
libmkl_blacs_intelmpi_lp64.so libmkl_intel_ilp64.so
libmkl_scalapack_ilp64.a locale
libmkl_blas95_ilp64.a libmkl_intel_lp64.a
libmkl_scalapack_ilp64.so
libmkl_blas95_lp64.a libmkl_intel_lp64.so
libmkl_scalapack_lp64.a
libmkl_cdft_core.a libmkl_intel_thread.a
libmkl_scalapack_lp64.so
You want libmkl_rt.so. This will resolve the provides problem.
Regards, Freddie.
Hi Freddie,
I set the environment variable “PYFR_XSMM_LIBRARY_PATH” in .bashrc
export PYFR_XSMM_LIBRARY_PATH= /FR/libxmm/lib/libxsmm.so
I also changed [backend-openmp] section according to your information.
[backend-openmp]
cc = icc
cblas = /opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64_lin/libmkl_rt.so
It works well! PyFR start to run without error. Thank you!
Best regards,
Tom