Compiling Numpy and Scipy with MKL

== Intel MKL 11.0 (updated Dec 2012) ==
The purpose of these instructions in to build and install 64-bit Numpy and Scipy, linking against the MKL libraries version 11.0. Here(http://software.intel.com/en-us/articles/numpyscipy-with-intel-mkl) is a slightly different procedure from Intel. Some of those steps were required on MKL 10.3 but are no longer required on 11.0.

Throughout, if you want to install 32-bit instead of 64-bit, replace `intel64` with `ia32` and `intelem` with `intel`. 

Download the following from the "Non-Commerical Software Development"
[downloads page.](http://software.intel.com/en-us/non-commercial-software-development)
"Intel Fortran Composer XE 2013 for Linux"
"Intel C++ Composer XE 2013 for Linux"
These packages contain the Intel C++ compiler, Fortran compiler, and
Math Kernel Libraries.

This page will automatically begin downloading a package containing both 32-bit and 64-bit software. I recommend cancelling this download, and then clicking the "intel64" alternative download.

Also download the source packages for numpy and scipy (linked above). I unpacked into:

{{{
~/dev/numpy-1.6.2
~/dev/scipy-0.11.0
}}}

Unpack the Intel Composer packages somewhere. Run both of the included `install.sh` with `sudo` privileges. Note the installation directory. It will probably be something like:

{{{
/opt/intel/composer_xe_2013
/opt/intel/composer_xe_2013.1.117
}}}

If these installations complete succesfully, you should be able to use the Intel C and Fortran compilers. I had to create a link manually.

{{{
ln -s /opt/intel/bin/ifort /usr/local/bin/ifort
ln -s /opt/intel/bin/icc /usr/local/bin/icc
}}}

Type `icc` and `ifort` at the command line to verify that they are installed.

Now switch to the numpy source directory `~/dev/numpy-1.6.2`. First create a `site.cfg` file.

{{{
cp site.cfg.example site.cfg
}}}

Now modify it manually to instruct it to build against MKL. Copy these lines in to the `site.cfg` file. There are some commented example files but I believe they are out of date.

{{{
[mkl]
library_dirs = /opt/intel/composer_xe_2013/mkl/lib/intel64
include_dirs = /opt/intel/composer_xe_2013/mkl/include
mkl_libs = mkl_rt
lapack_libs =
}}}

Next, modify the following file which controls which C compiler to use. We want it to use `icc` with some optimized flags. Open this file:

{{{
numpy/distutils/intelccompiler.py
}}}

Replace this line in IntelEM64TCCompiler.__init__:

{{{
self.cc_exe = 'icc -fPIC'
}}}

with this one:

{{{
self.cc_exe = 'icc -O3 -g -fPIC -fp-model strict -fomit-frame-pointer
-openmp -xhost'
}}}

I am not sure if the other lines also need to be modified.

The last step before compiling is setting some environment variables so that numpy knows about the MKL libraries and compilers.

{{{
source /opt/intel/composer_xe_2013/mkl/bin/mklvars.sh intel64
source /opt/intel/composer_xe_2013/bin/compilervars.sh intel64
source /opt/intel/composer_xe_2013.1.117/bin/iccvars.sh intel64
export LD_LIBRARY_PATH=/opt/intel/composer_xe_2013/mkl/lib/intel64:/opt/intel/composer_xe_2013/lib/intel64:$LD_LIBRARY_PATH
}}}

(You'll actually need to run that last command to set LD_LIBRARY_PATH every time you start a new terminal session (or put it in your start-up scripts). I am told that LD_LIBRARY_PATH is evil. I don't know how to avoid using it in this case.)

Now, we are ready to configure and build numpy. Make sure you are in the numpy source directory `~/dev/numpy-1.6.2`. If you think you might have a failed build, `rm -rf build` or `sudo rm -rf build` in that directory to force a re-build. Then configure:

{{{
python setup.py config --compiler=intelem build_clib
--compiler=intelem build_ext --compiler=intelem
}}}

And, if that succeeds, install. Note the installation directory.

{{{
sudo python setup.py install
}}}

You can mess around with the `--user` or `--prefix` flags if you want to install somewhere else.

Test that it worked by navigating to your home directory (or any directory except for the numpy source directory) and starting python or ipython. You might need to modify your path to remove any other numpy installations other than the one you just installed. Run this command to set your environment:

{{{
export LD_LIBRARY_PATH=/opt/intel/composer_xe_2013/mkl/lib/intel64:/opt/intel/composer_xe_2013/lib/intel64:$LD_LIBRARY_PATH
}}}

And then start ipython.

{{{
import numpy as np
print np.__file__ # shows path to installation
np.show_config()
}}}

You should see a bunch of stuff about MKL in the output from show_config.

Once you've made it this far, it's time to compile and install Scipy. Navigate to the scipy source directory `~/dev/scipy-0.11.0`. If you are in a new terminal session or if you're unsure, re-run the `source` and `export` commands above to set up the environment. Also make sure you can still run `icc` and `ifort`. Again, remove any build directory in the scipy directory.

Make certain that the only numpy installation on your Python path is the one you just installed (or at least, that the one you just installed is the first one on the path). If you are unsure, import numpy and check numpy.__file__. This version will be imported during the scipy build, and it had better be the one we just put all this work into.

Configure:

{{{
python setup.py config --compiler=intelem --fcompiler=intelem
build_clib --compiler=intelem --fcompiler=intelem build_ext
--compiler=intelem --fcompiler=intelem
}}}

And install:

{{{
sudo python setup.py install
}}}

I got a bizarre error on the last step about "unable to find libimf.so" even though this is on the path in LD_LIBRARY_PATH. My understanding is that this is some sort of bad interaction between scipy and sudo. I think this is the bad karma that results from the use of the evil LD_LIBRARY_PATH.
http://mail.scipy.org/pipermail/scipy-user/2008-November/018608.html

If this happens to you, run that last install command WITHOUT `sudo`, and install in your site directory (or anywhere on your Python path that you have write permission).

{{{
python setup.py --user install
}}}

To test, navigate to your home directory (or anywhere besides the numpy and scipy source directories) and fire up python or ipython. Again, make sure you've removed other installations from the Python path.

Run this command to set your environment:

{{{
export LD_LIBRARY_PATH=/opt/intel/composer_xe_2013/mkl/lib/intel64:/opt/intel/composer_xe_2013/lib/intel64:$LD_LIBRARY_PATH
}}}

And then start ipython.

{{{
import numpy as np
print np.__file__ # shows path to installation
np.show_config()
import scipy
print scipy.__file__
scipy.show_numpy_config()
scipy.show_config()
}}}

You should see stuff about MKL in all 3 show_config() outputs.

You're done! One last time: remember to always set your LD_LIBRARY_PATH before starting a python or ipython session in a new terminal.
Comments