Link numpy against PyPI mkl - python-3.x

So I would like to build numpy against MKL BLAS on Ubuntu 20.04.
I have no trouble building and linking against system installed mkl (from e.g. the Ubuntu intel-mkl package). As documented in site.cfg.example I created a copy site.cfg next to it and filled in the relevant information
[mkl]
library_dirs = /path/to/my/local/mkl/lib
include_dirs = /path/to/my/local/mkl/include
libraries = mkl_rt
I can create a wheel with python -m build --wheel that is linked against that MKL, I can install it fine with pip install my_numpy.whl and after import numpy, numpy.show_config() shows the expected info regarding mkl.
Now what I would like to do is to be independent from
system installed mkl
absolute paths in library_dirs and include_dirs
All the necessary library and include files for building and linking against MKL are available in the PyPI packages mkl and mkl-include. I would like to configure a build process, where these two packages are downloaded, built and linked against and also added to the requirements. Now I understand running python3 -m build creates a separate venv and preinstalls any packages listed in pyproject.toml. Here my edited version with mkl and mkl-include added at the bottom
[build-system]
# Minimum requirements for the build system to execute.
requires = [
"packaging==20.5; platform_machine=='arm64'", # macos M1
"setuptools<49.2.0",
"wheel==0.36.2",
"Cython>=0.29.24,<3.0", # Note: keep in sync with tools/cythonize.py
"mkl-include<2021.2",
"mkl<2021.2"
]
I am stuck at what to put into site.cfg under library_dirs and include_dirs to use the files from the packages installed by the build venv. Is there a way to include the current venv's prefix into those paths? If yes, how? Do the numpy.distutils offer anything in that regard?
Or do I have to approach this problem differently? E.g. is there a way to build numpy and create a .whl, but without explicitly linking, s.t. linking can be delayed until actual installation time of the built wheel, where the actual MKL runtime libraries are found (in this case /path/to/my/current/venv/lib) and linked against? Is that even possible with a wheel?
Thanks for your help!

Related

Cabal: missing dependency on foreign C library

I have been trying to install the tensorflow/haskell package on my Windows machine.
However, both while installing with stack and cabal, I keep running into this error:
Cabal-simple_Z6RU0evB_1.24.2.0_ghc-8.0.2.exe: Missing dependency on a foreign
library:
* Missing C library: tensorflow
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.
According to the Tensorflow website: https://www.tensorflow.org/install/install_c
, the C package for Tensorflow only runs on Unix machines. I am not sure if this is a dead end or if there is some workaround.
I am not experienced with linking libraries or with cabal, and would appreciate some assistance.
Using Cabal, I've included the following .cabal file in my directory:
cabal-version: >= 1.2
Executable hax
Includes: c_api.h
Extra-libraries: tensorflow
but I still receive the errors: cabal.exe: Missing dependency on a foreign library: * Missing C library: tensorflow The exception was: user error (Using 'build-type: Custom' but there is no Setup.hs or Setup.lhs script.)
You write "According to the Tensorflow website: https://www.tensorflow.org/install/install_c , the C package for Tensorflow only runs on Unix machines. I am not sure if this is a dead end or if there is some workaround."
Indeed, if you don't have the tensorflow C package installed on your machine -- because it won't run on it, then you can't find it to bind to it.
There is no work around for that.

Problems installing gtk2hs on windows

I have decided to try to install gtk2hs on my Windows 8 (64 bit) machine. I am relying on the lastest Haskell Platform distribution.
So, I go on the wiki, and follow the instructions. I get gtk+ 32-bit distribution, version 2.24, in the all in one bundle. I unpack it correclty, set up PATH to point to gtk/bin, and create INCLUDE and LIB variables to point respectively to gtk/include and gtk/lib (this step does not seem necessary, but it can't hurt!)
I then verify all is in order by invoking
pkg-config --modversion gtk+-2.0
and the terminal happly chirps back
2.24.10
Great!
Then, I run cabal install
gtk2hs-buildtools
and that also goes well.
But then, when I run cabal install gtk, it complains that glib fails to install, because it can't find some C dependencies. I try to isolate the problem by running
cabal install glib
and get the following:
C:\Users\Federico>cabal install glib
Resolving dependencies...
Configuring glib-0.13.1.0...
Failed to install glib-0.13.1.0
Last 10 lines of the build log ( C:\Users\Federico\AppData\Roaming\cabal\logs\gl
ib-0.13.1.0.log ):
[1 of 2] Compiling Gtk2HsSetup ( Gtk2HsSetup.hs, dist\setup-wrapper\Gtk2HsS
etup.o )
[2 of 2] Compiling Main ( SetupMain.hs, dist\setup-wrapper\Main.o )
Linking dist/setup-wrapper\setup.exe ...
Configuring glib-0.13.1.0...
setup.exe: Missing dependencies on foreign libraries:
* Missing C libraries: gobject-2.0, gthread-2.0, glib-2.0, intl
This problem can usually be solved by installing the system packages that
provide these libraries (you may need the "-dev" versions). If the libraries
are already installed but in a non-standard location then you can use the
flags --extra-include-dirs= and --extra-lib-dirs= to specify where they are.
cabal: Error: some packages failed to install:
glib-0.13.1.0 failed during the configure step. The exception was:
ExitFailure 1
But this is utter crap! I KNOW the libraries are there. I can see them, in the gtk/bin folder! I re-check my path, and all is good. They are visible, in path. so is include, and lib. All of it. And yet, cabal complains it can't find them.
But this isn't the scariest bit yet...I download the 64-bit of the library, version 2.16...simply swap the two gtk folders...and now, cabal happly builds the whole thing! Of course, it's unsuable, because ghc on the plaftorm is 32-bit and the libraries are 64-bit, so ghc fails to link against it...but at least cabal was pleased.
I then remove glib, cairo, pango, gtk etc, and go about trying installing again, 2.16, 32-bit. And surely, cabal complains about C dependencies missing.
At this point, befuddled by this kafkian situation, I turn to the wisdom of stack overflow!
Is cabal trolling me? Are the error messages misleading? Or am I missing something obvious?
Thanks to all for the help!
If you are using 64 bit version of GHC, you want to use 64 bit version of gtk as well.
You can download it on this link. Scroll down a bit and download 2.x (not 3.x) all in one bundle, don't download 2.16:
http://www.gtk.org/download/win64.php
Above is no longer an option. You can try installing MSYS and than using MSYS console with pacman package manager to install gtk dev library on windows (I haven't tried it yet though). This pretty much sucks in my opinion.

Building Pango with Cairo support

I'm attempting to build a new version of wireshark 1.10 on a RHEL5 box with the intent of making an RPM so that customers that I support can install it. Possibly others if I can find a place to host it.
WS1.10 depends on GTK+ as well as several other libraries that aren't supported by the available rpm packages. Therefore, I have to build them by hand. GTK+ needs pango(1.24.5) with cairo(1.8.8) support.
After having built and installed a newer version of cairo from source, a ./configure of pango gives the following error
checking for CAIRO... yes
checking which cairo font backends could be used... none
configure: Disabling cairo support
and then at the end ./configure displays
configuration:
backends: X
I can verify that Cairo is installed by looking at the pkg_config in my /usr/local/lib.
I've even taken a look at the pango config.log. But I don't see anything that is any different then the messages I copypasta'ed above.
What are some places I should be looking to figure out a less terse error message? I'm kind of at a loss troubleshooting this.
I encountered this issue today and solved it by installing HARFBUZZ, which is mentioned in the README file of Pango. After installation of Harfbuzz we have access to FreeType. So follow this procedure:
Build and install FreeType
Build and install fontconfig
Build and install Cairo (after ./configure please make sure freetype and fontconfig are found)
Build and install Harfbuzz
Build and install Pango
In case of you still have problems even though you are following the guideline of Xiao Jia, take care to look into config.log. It will help you to find out why the package does not find particular components on your system. In my case the fontconfig-devel and freetype-devel package from default repository of my distro was too old.
I could manage it by installing the whole stack into a location outside of /usr (not that difficult as it sounds) by adding configuration file to /etc/ld.so.conf.d and a environment expanding script to /etc/profile.d.

How to install scikit-learn-0.14.1 with atlas at non-standard location

In Ubuntu 12.04 under python 2.7.3 I am
trying to install scikit-learn-0.14.1 using the
"test the package before installing". The first step
(python setup.py build_ext --inplace) since to finish OK
See the output at [ https://gist.github.com/anonymous/7043418 ]
The testing step, however (nosetests sklearn/), things are not so nice.
It ended with 22 errors and 9 tests failed.
See the output at [ https://gist.github.com/anonymous/7044411 ]
If I am right, the problem since to be related to the atlas library.
I am using my own build of the libraries blas-lapack-atlas, the
directory at which they are installed (/home/myacct/myProg/NumLibs64b/lib)
is listed at the
beginning of the building process. However, no where the compiling options
of gcc/g++ includes -llapack -lcblas -lf77blas -latlas
Only a mention of '-DNO_ATLAS_INFO=-1 is listed during the building process.
Does this option means not to use atlas?
Numpy and Scipy were build successfully using the same
blas-lapack-atlas libraries (via the corresponding STATIC versions).
So I guess the question is how to modify setup.py of scikit-learn-0.14.1
in order to build it using
a non-standard location for the STATIC libraries blas-lapack-atlas?
Regards,
Sergio

Building mysql++ libraries for powerpc

I have an embedded application written in C++ (running on a PowerPC in linux environment) which accesses an external database. So my application needs mysql++ libraries which needs to be built using a powerpc compiler. Before building the libraries, I wanted to build mysql++ libraries for linux on my Ubuntu VM to check the procedure. I downloaded the latest package from Official Website. Then I followed the steps mentioned.
Ran ./configure from the root directory, I got the below error.
*checking for MySQL library directory... configure: error: Didn't find mysqlclient library in '/usr/lib64 /usr/lib /usr/lib64/mysql /usr/lib/mysql /usr/local/lib64 /usr/local/lib /usr/local/lib/mysql /usr/local/mysql/lib /usr/local/mysql/lib/mysql /usr/mysql/lib/mysql /opt/mysql/lib /opt/mysql/lib/mysql /sw/lib /sw/lib/mysql'*
I tried locate mysqlclient and did not find any references, hence I followed instructions given in other forums and tried to install the libmysqlclient15-dev using the below command.
sudo apt-get install libmysqlclient15-dev
The output is as shown below
Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'libmysqlclient-dev' instead of 'libmysqlclient15-dev'
libmysqlclient-dev is already the newest version.
The following packages were automatically installed and are no longer required:
libcommons-collections3-java junit4 libecj-java libdb5.1-java libasm3-java libgcj-bc gcj-4.6-jre-lib libgnomeui-common libcommons-el-java
junit linux-headers-3.2.0-32 linux-headers-3.2.0-29 libcommons-compress-java libregexp-java libdb-java libswt-cairo-gtk-3-jni libjasper-java
libbonoboui2-common libbonoboui2-0 libdb5.1-java-gcj libcommons-httpclient-java libservlet2.4-java liblucene2-java libswt-gtk-3-java
libcommons-cli-java libslf4j-java libgcj12 libxml-commons-external-java libswt-webkit-gtk-3-jni linux-headers-3.2.0-29-generic-pae
libswt-gtk-3-jni ant gcj-4.6-base libcommons-logging-java default-jdk libswt-glx-gtk-3-jni libcommons-codec-java jarwrapper
libequinox-osgi-java libgcj-common libapache-pom-java libgnomeui-0 libjetty-java libjline-java libxerces2-java sat4j
libcommons-beanutils-java libdb-je-java fastjar libcommons-digester-java libcommons-parent-java libhamcrest-java libjtidy-java
libxml-commons-resolver1.1-java libicu4j-java linux-headers-3.2.0-32-generic-pae libicu4j-4.4-java libwebkitgtk-1.0-common
libcommons-lang-java libwebkitgtk-1.0-0 libjavascriptcoregtk-1.0-0 libjsch-java ant-optional libswt-gnome-gtk-3-jni
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
After this I tried to locate libmysqlclient and I still couldn't find the libraries (.so files). I tried ./configure again and it failed in the same location with the same error message. How can I install libmysqlclient ?
Once this is solved I will rebuild the mysql++ library using my toolchain for powerpc. Has anyone tried this ? To build mysql++ library I need mysqlclient, in the earlier example I would install it by executing the command apt-get, but the libraries would be compiled using a gnu compiler ? How to cross compile mysqlclient for powerpc ? I guess I need to do the below
Build mysqlclient for powerpc.
Use those libraries and build mysql++ libraries for powerpc.
Use mysql++ and mysqlclient libraries in my application and compile for powerpc.
I am new to linux and databases.
I finally solved this problem. I did the below.
Downloaded the Connector/C (libmysql) source files from here
Cross compiled it for powerpc, below is the command given to cmake
cmake -DCMAKE_INSTALL_PREFIX="~/mysqlclient_C/lib" -DCMAKE_C_COMPILER="ppc_4xx-gcc" -DCMAKE_C_FLAGS="-I/opt/ELDK/4.2/ppc_4xx/usr/include/" -DCMAKE_CXX_COMPILER="ppc_4xx-g++" -DCMAKE_CXX_FLAGS="-I/opt/ELDK/4.2/ppc_4xx/usr/include/" -DCMAKE_EXE_LINKER_FLAGS="-lm"
Then a simple make generated the libmysql libraries.
Downloaded mysql++ source files from here
Cross compiled it for powerpc, below is my configure command
./configure --target=powerpc-linux --host=powerpc-linux --prefix="$HOME/mysql++/lib" --enable-thread-check --with-mysql="$HOME/mysqlclient_C/lib" CC=ppc_4xx-gcc CXX=ppc_4xx-g++ LDFLAGS=-lm CFFLAGS="-I/opt/ELDK/4.2/ppc_4xx/usr/include" CXXFLAGS="-I/opt/ELDK/4.2/ppc_4xx/usr/include"
The mysql++ libraries is generated now.
I have not tested the built libraries yet.

Resources