Spark doesn't find BLAS in MKL dll - apache-spark

I'm working on IntelliJ and specified this parameter to my JVM :
-Dcom.github.fommil.netlib.BLAS=mkl_rt.dll (my mkl folder is in the Path)
However I still have the following warning :
WARN BLAS: Failed to load implementation from: mkl_rt.dll
Any help ?

I finally solved this issue, here's the complete step to do make it work on intelliJ Idea on Windows :
First create an SBT project and make sure to put the following line in build.SBT :
libraryDependencies ++= Seq("com.github.fommil.netlib" % "all" % "1.1.1" pomOnly())
Refresh the project, after that you should have the libraries available. If that doesn't work for some reason, you can go here : http://repo1.maven.org/maven2/com/github/fommil/netlib/ and download the necessary resources for your system directly.
Copy your mkl_rt.dll twice and rename the copies libblas3.dll and liblapack3.dll. Make sure your folders containing all the Dll is in the PATH environment variable.
Finally, go to Run -> Edit configuration and in the VM options put :
-Dcom.github.fommil.netlib.BLAS=mkl_rt.dll

Related

unresolved dependency: com.eed3si9n#sbt-assembly;0.13.0: not found

Did lots of search, saw many people having the similar issue and tried various suggested solution. None worked.
Can someone help me?
resolvers += Resolver.url("bintray-sbt-plugins", url("http://dl.bintray.com/sbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")
The file is inside the project folder.
Instead of 0.13.0 version, I used 0.14.0 version.
I fixed this by adding POM file which I downloaded from
https://dl.bintray.com/sbt/sbt-plugin-releases/com.eed3si9n/sbt-assembly/scala_2.10/sbt_0.13/0.14.4/ivys/
to my local ivy folder under below location .ivy/local ( if not present, create the local folder).
once it was there I ran the build and it downloaded the jar.
You need to add [root_dir]/project/plugins.sbt file with the following content:
// packager
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
Event better - don't use sbt-assembly at all! Flat-jars cause conflicts during merging which need to be resolved with assemblyMergeStrategy.
Use the binary distribution format plugin that sbt offers which enables you to distribute in binary script, dmg, msi and tar.gz.
Check out sbt-native-packager

Android-NDK: Unsatisfied Link Error : Cannot load library: soinfo_relocate: cannot locate symbol "tcgetattr"

I am building the app available at https://github.com/thibautd/android-serialport-api/tree/master/android-serialport-api/project using Android NDK.
What have I done until now:
Build the project using gradle version 'com.android.tools.build:gradle-experimental:0.7.0-alpha1'
I am building this on android-17 and running on Android device with 4.2. I have also tried android-19
Build goes through without any errors but at runtime I get the exception Cannot load library: soinfo_relocate: cannot locate symbol "tcgetattr"
I have tried several options to get this working but all of them fail
Options I have tried
Setting APP-PLATFORM like mentioned in Cannot locate symbol 'tcgetattr' referenced by "libcrypto.so"
Using $(PREBUILT_SHARED_LIBRARY) by including the .so file avialable in the same github location as mentioned in How can i Link prebuilt shared Library to Android NDK project?
Tried setting LOCAL_EXPORT_C_INCLUDES in both option 1) and 2) above as mentioned in the same post How can i Link prebuilt shared Library to Android NDK project?
Also tried option mentioned at Cannot load library: reloc_library[1285]: cannot locate 'rand'
Put all the required include files directly in the project under /jni and tried building the project.
My project compiles absolutely fine. When testing I have always deleted the contents in build folder and rebuilt the project. To make sure the .so files are indeed created properly.
In all the cases I end up in exactly the same error soinfo_relocate: cannot locate symbol "tcgetattr"
I think it is because of the fact that some of the includes like unistd.h are not correctly referenced in runtime but I am not able to figure out how to get this working.

sbt, ivy, offline work, and weirdness

I'm trying to work on an sbt project offline (again). Things almost seem to be ok, but there are strange things that I'm baffled by. Here's what I'm noticing:
I've created an empty sbt project and am considering the following dependencies in build.sbt:
name := "sbtSand"
version := "1.0"
scalaVersion := "2.11.7"
libraryDependencies ++= Seq(
"joda-time" % "joda-time" % "2.9.1",
"org.apache.spark" %% "spark-core" % "1.5.2"
)
I've built the project while online, and can see all the packages in [userhome]/.ivy2/cache. The project builds fine. I then turn off wifi, sbt clean and attempt to build. The build fails. I comment out the spark dependency (keeping the joda-time one). Still offline, I run sbt compile. The project builds fine. I put the spark dependency back in, and sbt clean. It again fails to build. I get back online. I can build again.
The sbt output for the failed builds are like: https://gist.github.com/ashic/9e5ebc39ff4eb8c41ffb
The key part of it is:
[info] Resolving org.apache.hadoop#hadoop-mapreduce-client-app;2.2.0 ...
[warn] Host repo1.maven.org not found. url=https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-app/2.2.0/hadoop-mapreduce-client-app-2.2.0.pom
[info] You probably access the destination server through a proxy server that is not well configured.
It's interesting that sbt is managing to use the joda-time from ivy cache, but for the spark-core package (or rather its dependency) it wants to reach out to the internet and fails the build. Could anybody please help me understand this, and what I can do so that I can get this to work while fully offline?
It seems the issue is resolved in 0.13.9. I was using 0.13.8. [The 0.13.9 msi for windows seemed to give me 0.13.8, while the 0.13.9.2 msi installed the right version. Existing projects need updating manually to 0.13.9 in build properties.]

Unable to execute dex: Multiple dex files define Lcom/actionbarsherlock/R$attr;

i use ActionBarLibrary in a library, that I use in a my app
i got this message when i try to compil
[2013-01-28 16:09:46 - Dex Loader] Unable to execute dex: Multiple dex files define Lcom/actionbarsherlock/R$attr;
[2013-01-28 16:09:46 - comfree] Conversion to Dalvik format failed: Unable to execute dex: Multiple dex files define Lcom/actionbarsherlock/R$attr;
As asked here, I was facing the exact same error yesterday. I pinpointed the cause of the error to /bin/classes. If I deleted the /classes dir before running the app, the app would compile and run. Of course, when I tried to actually export the app, the /classes dir would come back and along with it the error.
The solution to my problem was to install the latest version of Eclipse. You will then need to install the latest version of the ADT but all the Android versions you have downloaded earlier for the other Eclipse installation will be detected and used for this one as well. When I export or run the app through Eclipse, the /classes folder does not appear now.
right click on project to go project property>>java Build path>>choose Order and Export and then Unselect all except android4.0 or what every you have then clean and build your project ,now error not come again..
The solution to my problem was that I deleted the jar files which I copy pasted in project's lib folder. I explicitly imported external jars in build path

InstallShield 2011 Standalone Builder does not include SetupPrerequisites required for .NET 3.5

The 2011 InstallShield Standalone Builder does not include the SetupPrerequisites required for .NET 3.5; when building with the Full installation, it completes successfully, but when building with the SAB, it gives the following set of errors:
ISDEV : error -1007: Cannot copy source 'c:\Program Files\InstallShield\2011 SAB\SetupPrerequisites\Microsoft .net\3.5\Full\dotnetfx35.exe' to target '[path]\Release\DiskImages\DISK1\ISSetupPrerequisites{2B692A0F-7172-44B2-B0CB-9991078AFD31}\dotnetfx35.exe'
ISDEV : error -5054: Could not determine the size of the file "[path]\Release\DiskImages\DISK1\ISSetupPrerequisites{2B692A0F-7172-44B2-B0CB-9991078AFD31}\dotnetfx35.exe"
ISDEV : error -1007: Cannot copy source 'c:\Program Files\InstallShield\2011 SAB\SetupPrerequisites\Microsoft .net\3.5\Full\Helper.exe' to target '[path]\Release\DiskImages\DISK1\ISSetupPrerequisites{2B692A0F-7172-44B2-B0CB-9991078AFD31}\Helper.exe'
ISDEV : error -5054: Could not determine the size of the file "[path]\Release\DiskImages\DISK1\ISSetupPrerequisites{2B692A0F-7172-44B2-B0CB-9991078AFD31}\Helper.exe"
It is somewhat strange that after not being able to copy (from SRC to DEST), it still tries to determine the file size of DEST!
But anyway, the root cause is that the "Microsoft .net" folder under SetupPrerequisites in the install directory (c:\Program Files\InstallShield\2011 SAB) is missing. It should have a folder "3.5", which should have a folder "Full", which should contain two files: dotnetfx35.exe and Helper.exe.
We have fixed this in our SAB installations by copying these files from the Full installation to the SAB installation.
Actually, we already had an InstallShield folder with the ISM in it, so we just created a SetupPrerequisites folder under there, put the same structure as should have been installed, and then added logic to the build such that if it sees this folder, it will copy everything in it down to the SAB install folder so that the build will succeed.
I'm answering my own question in the above paragraph; I wanted to put this out there for anyone else who might experience this issue.
Your observations are correct and I don't like the way InstallShield works ( by default ) as it encourages the magical (build) machine antipattern.
Instead, I prefer to edit the PRQ files and the ISM references to the PRQs ( ISSetupPrerequisites table, ISBuildSourcePath column ) and make everything use relative paths based on the ISPROJECTDIR path variable. I check then check all the PRQ files and redist files into my source control so that when my build pulls the tree everything can be found. This pattern can be made more complex if you are reusing redists across multiple builds but the concept is the same.

Resources