# PyFR on Xeon Phi **URL:** \ **Category:** General\ **Created:** [6 June 2017 05:24 UTC](https://pyfr.discourse.group/t/pyfr-on-xeon-phi/165 "2017-06-06T05:24:19Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Tom\_Vilsack](https://avatars.discourse-cdn.com/v4/letter/t/b9e5f3/32.png) [@Tom\_Vilsack](https://pyfr.discourse.group/u/Tom_Vilsack)\ **Post date:** [6 June 2017 05:24 UTC](https://pyfr.discourse.group/t/pyfr-on-xeon-phi/165/1 "2017-06-06T05:24:19Z")
Dear All, I’m trying to run PyFR on Xeon Phi 7250 (Knight Landing). There is only one Xeon Phi processor, and no CPU on my computing node. Therefore, I log in to the Xeon Phi from front-end node via ssh, and execute command on Xeon Phi directly. In this case, how I can write “backend-mic” section in .ini file? Could you tell me sample description of “device-id” and “mkl-root” parameters? I could run PyFR using “backend-openmp” settings on Xeon Phi. However, I think simulation speed will be faster if I can run PyFR with mic native functions. Best regards, Tom ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [6 June 2017 09:00 UTC](https://pyfr.discourse.group/t/pyfr-on-xeon-phi/165/2 "2017-06-06T09:00:32Z")
Hi Tom, > I'm trying to run PyFR on Xeon Phi 7250 (Knight Landing). There is only > one Xeon Phi processor, and no CPU on my computing node. Therefore, I > log in to the Xeon Phi from front-end node via ssh, and execute command > on Xeon Phi directly. > > In this case, how I can write "backend-mic" section in .ini file? Could > you tell me sample description of "device-id" and "mkl-root" parameters? > > I could run PyFR using "backend-openmp" settings on Xeon Phi. However, I > think simulation speed will be faster if I can run PyFR with mic native > functions. For KNL you want to be using the OpenMP backend. The key here is to make sure that you get the number of OpenMP threads correct, have MKL available, use the ICC compiler, and have libxsmm installed. Together, these should result in rather good performance for non-pathological test cases. The MIC backend is for the legacy Knights Corner. Regards, Freddie. ---
**Author:** ![Tom\_Vilsack](https://avatars.discourse-cdn.com/v4/letter/t/b9e5f3/32.png) [@Tom\_Vilsack](https://pyfr.discourse.group/u/Tom_Vilsack)\ **Post date:** [6 June 2017 14:47 UTC](https://pyfr.discourse.group/t/pyfr-on-xeon-phi/165/3 "2017-06-06T14:47:17Z")
Hi Freddie, Thank you for your reply. I have installed libxsmm according to commands as described below. make STATIC=0 BLAS=0 make PREFIX= STATIC=0 BLAS=0 install The make process finished without error. Then, I set parameter in “backend-openmp” section in ini file. [backend-openmp] cc = icc cblas = /lib/libxsmm.so Are the setting parameters correct? Xeon Phi 7250 have 68 physical cores, and each core can run 4 threads. Therefore, I set $OMP\_NUM\_THREADS=272. Could you tell me meanings of “gimmik-max-nnz”, “libxsmm-block-sz” and “libxsmm-max-sz” in “backend-openmp” section in ini file? Best regards, Tom ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [6 June 2017 15:00 UTC](https://pyfr.discourse.group/t/pyfr-on-xeon-phi/165/4 "2017-06-06T15:00:10Z")
Hi Tom, > Thank you for your reply. > > I have installed libxsmm according to commands as described below. > >   make STATIC=0 BLAS=0 >   make PREFIX=\ STATIC=0 BLAS=0 install > > The make process finished without error. Then, I set parameter in > "backend-openmp" section in ini file. > >   [backend-openmp] >   cc = icc >   cblas = \/lib/libxsmm.so So libxsmm is not a cblas library (that is MKL). All you need to do is ensure that libxsmm is somewhere where PyFR can find it. If need be you can point PyFR to the library by doing     export PYFR\_XSMM\_LIBRARY\_PATH=/path/here/to/lib.so > Are the setting parameters correct? Xeon Phi 7250 have 68 physical > cores, and each core can run 4 threads. Therefore, I set > $OMP\_NUM\_THREADS=272. For best performance you want one thread per core. > Could you tell me meanings of "gimmik-max-nnz", "libxsmm-block-sz" and > "libxsmm-max-sz" in "backend-openmp" section in ini file? The first parameter decides the cut-off point at which we will no longer use the GiMMiK matrix multiplication library. The second parameter controls the block size for libxsmm, although performance does not appear to be too sensitive to its value. The final parameter decides the cut-off point at which we will no longer use libxsmm for matrix multiplications. The hierarchy is: 1. libxsmm (if available and size \< libxsmm-max-sz) 2. GiMMiK (else if nonzeros \< gimmik-max-nnz) 3. cblas (else) The defaults in PyFR are not too bad, but you may be able to get 20-30% out for certain test cases by playing around with these parameters. The key is to ensure that libxsmm is available, this is especially important at lower polynomial orders. Regards, Freddie. ---
**Author:** ![Tom\_Vilsack](https://avatars.discourse-cdn.com/v4/letter/t/b9e5f3/32.png) [@Tom\_Vilsack](https://pyfr.discourse.group/u/Tom_Vilsack)\ **Post date:** [7 June 2017 13:32 UTC](https://pyfr.discourse.group/t/pyfr-on-xeon-phi/165/5 "2017-06-07T13:32:23Z")
Hi Freddie, Thank you for the detail information. I set the environment variable “PYFR\_XSMM\_LIBRARY\_PATH” in .bashrc export PYFR\_XSMM\_LIBRARY\_PATH=$HOME/FR/libxmm/lib In libxmm/lib, there are dynamic libraries like libxsmm\*.so. Then, I set “[backend-openmp]” section in .ini file as described below. [backend-openmp] cc = icc cblas = /opt/intel/compilers\_and\_libraries\_2017.2.174/linux/mkl/lib/intel64\_lin\_mic/libmkl\_rt.so I got an error “KeyError: “‘mul’ has no providers”” when I execute PyFR. The error could be related to wrong cblas library pass. In mkl lib directory, there are many libraries. What dynamic library I should set in cblas section? libmkl\_ao\_worker.so libmkl\_cdft\_core.so libmkl\_intel\_thread.so libmkl\_sequential.a libmkl\_blacs\_intelmpi\_ilp64.a libmkl\_core.a libmkl\_lapack95\_ilp64.a libmkl\_sequential.so libmkl\_blacs\_intelmpi\_ilp64.so libmkl\_core.so libmkl\_lapack95\_lp64.a libmkl\_tbb\_thread.a libmkl\_blacs\_intelmpi\_lp64.a libmkl\_intel\_ilp64.a libmkl\_rt.so libmkl\_tbb\_thread.so libmkl\_blacs\_intelmpi\_lp64.so libmkl\_intel\_ilp64.so libmkl\_scalapack\_ilp64.a locale libmkl\_blas95\_ilp64.a libmkl\_intel\_lp64.a libmkl\_scalapack\_ilp64.so libmkl\_blas95\_lp64.a libmkl\_intel\_lp64.so libmkl\_scalapack\_lp64.a libmkl\_cdft\_core.a libmkl\_intel\_thread.a libmkl\_scalapack\_lp64.so Best regards, Tom ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [7 June 2017 19:25 UTC](https://pyfr.discourse.group/t/pyfr-on-xeon-phi/165/6 "2017-06-07T19:25:31Z")
Hi Tom, > Thank you for the detail information. > > I set the environment variable "PYFR\_XSMM\_LIBRARY\_PATH" in .bashrc The path needs to be the absolute path to the library itself as opposed to the directory in which it is located. >   export PYFR\_XSMM\_LIBRARY\_PATH=$HOME/FR/libxmm/lib > > In libxmm/lib, there are dynamic libraries like libxsmm\*.so. > > Then, I set "[backend-openmp]" section in .ini file as described below. > >   [backend-openmp] >   cc = icc >   cblas = > /opt/intel/compilers\_and\_libraries\_2017.2.174/linux/mkl/lib/intel64\_lin\_mic/libmkl\_rt.so > > I got an error "KeyError: "'mul' has no providers"" when I execute PyFR. > The error could be related to wrong cblas library pass. In mkl lib > directory, there are many libraries. What dynamic library I should set > in cblas section? > >   libmkl\_ao\_worker.so libmkl\_cdft\_core.so > libmkl\_intel\_thread.so libmkl\_sequential.a >   libmkl\_blacs\_intelmpi\_ilp64.a libmkl\_core.a > libmkl\_lapack95\_ilp64.a libmkl\_sequential.so >   libmkl\_blacs\_intelmpi\_ilp64.so libmkl\_core.so > libmkl\_lapack95\_lp64.a libmkl\_tbb\_thread.a >   libmkl\_blacs\_intelmpi\_lp64.a libmkl\_intel\_ilp64.a > libmkl\_rt.so libmkl\_tbb\_thread.so >   libmkl\_blacs\_intelmpi\_lp64.so libmkl\_intel\_ilp64.so > libmkl\_scalapack\_ilp64.a locale >   libmkl\_blas95\_ilp64.a libmkl\_intel\_lp64.a > libmkl\_scalapack\_ilp64.so >   libmkl\_blas95\_lp64.a libmkl\_intel\_lp64.so > libmkl\_scalapack\_lp64.a >   libmkl\_cdft\_core.a libmkl\_intel\_thread.a > libmkl\_scalapack\_lp64.so You want libmkl\_rt.so. This will resolve the provides problem. Regards, Freddie. ---
**Author:** ![Tom\_Vilsack](https://avatars.discourse-cdn.com/v4/letter/t/b9e5f3/32.png) [@Tom\_Vilsack](https://pyfr.discourse.group/u/Tom_Vilsack)\ **Post date:** [9 June 2017 13:05 UTC](https://pyfr.discourse.group/t/pyfr-on-xeon-phi/165/7 "2017-06-09T13:05:19Z")
Hi Freddie, I set the environment variable “PYFR\_XSMM\_LIBRARY\_PATH” in .bashrc export PYFR\_XSMM\_LIBRARY\_PATH= /FR/libxmm/lib/libxsmm.so I also changed [backend-openmp] section according to your information. [backend-openmp] cc = icc cblas = /opt/intel/compilers\_and\_libraries\_2017.2.174/linux/mkl/lib/intel64\_lin/libmkl\_rt.so It works well! PyFR start to run without error. Thank you! Best regards, Tom