Spark netlib-java openBLAS - apache-spark

Question

I am using word2vec in mllib and I want to improve the speed of linear operations in word2vec by configuring OpenBLAS.
I configured OpenBLAS on the server by manually compiling OpenBLAS with the following command.
git clone https://github.com/xianyi/OpenBlas.git
cd OpenBLAS
make USE_THREAD=0 -j16
make PREFIX=/usr/local install
sudo update-alternatives --install /usr/lib/libblas.so libblas.so /usr/local/lib/libopenblas.so 5
sudo update-alternatives --install /usr/lib/libblas.so.3 libblas.so.3 /usr/local/lib/libopenblas.so 5
sudo update-alternatives --install /usr/lib/liblapack.so liblapack.so /usr/local/lib/libopenblas.so 5
sudo update-alternatives --install /usr/lib/liblapack.so.3 liblapack.so.3 /usr/local/lib/libopenblas.so 5
I compiled and ran the following code to make sure I had OpenBLAS configured correctly:
#include <cblas.h>
#include <stdio.h>
int main() {
double A[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
double B[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
double C[9] = {.5,.5,.5,.5,.5,.5,.5,.5,.5};
cblas_dgemm(CblasColMajor, CblasNoTrans, CblasTrans,3,3,2,1,A, 3, B, 3,2,C,3);
for(int i = 0; i < 9; i++)
printf("%lf ", C[i]);
printf("\n");
return 0;
}
Use the following command to compile:
gcc test.c -lblas
The program compiles and runs well and produces the correct results.
I have included the netlib-java library in my spark project build file pom.xml as follows:
<dependency>
<groupId>com.github.fommil.netlib</groupId>
<artifactId>all</artifactId>
<version>1.1.2</version>
<type>pom</type>
</dependency>
I then ran my spark project locally and still got the following warning:
2022-11-09 02:34:48 WARN BLAS:61 - Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
2022-11-09 02:34:48 WARN BLAS:61 - Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
2022-11-09 02:34:48 WARN APP:66 - com.github.fommil.netlib.F2jBLAS
I would like to know what might be wrong? My server configuration is as follows:
Ubuntu 18.04.1 LTS
spark-2.3.2
Intel(R) Xeon(R) W-2145 CPU # 3.70GHz
I am using word2vec in mllib and I want to improve the speed of linear operations in word2vec by configuring OpenBLAS.
I configured OpenBLAS on the server by manually compiling OpenBLAS with the following command.
git clone https://github.com/xianyi/OpenBlas.git
cd OpenBLAS
make USE_THREAD=0 -j16
make PREFIX=/usr/local install
sudo update-alternatives --install /usr/lib/libblas.so libblas.so /usr/local/lib/libopenblas.so 5
sudo update-alternatives --install /usr/lib/libblas.so.3 libblas.so.3 /usr/local/lib/libopenblas.so 5
sudo update-alternatives --install /usr/lib/liblapack.so liblapack.so /usr/local/lib/libopenblas.so 5
sudo update-alternatives --install /usr/lib/liblapack.so.3 liblapack.so.3 /usr/local/lib/libopenblas.so 5
I compiled and ran the following code to make sure I had OpenBLAS configured correctly:
#include <cblas.h>
#include <stdio.h>
int main() {
double A[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
double B[6] = {1.0,2.0,1.0,-3.0,4.0,-1.0};
double C[9] = {.5,.5,.5,.5,.5,.5,.5,.5,.5};
cblas_dgemm(CblasColMajor, CblasNoTrans, CblasTrans,3,3,2,1,A, 3, B, 3,2,C,3);
for(int i = 0; i < 9; i++)
printf("%lf ", C[i]);
printf("\n");
return 0;
}
Use the following command to compile:
gcc test.c -lblas
The program compiles and runs well and produces the correct results.
I have included the netlib-java library in my spark project build file pom.xml as follows:
<dependency>
<groupId>com.github.fommil.netlib</groupId>
<artifactId>all</artifactId>
<version>1.1.2</version>
<type>pom</type>
</dependency>
I then ran my spark project locally and still got the following warning:
2022-11-09 02:34:48 WARN BLAS:61 - Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
2022-11-09 02:34:48 WARN BLAS:61 - Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
2022-11-09 02:34:48 WARN APP:66 - com.github.fommil.netlib.F2jBLAS
I would like to know what might be wrong? My server configuration is as follows:
Ubuntu 18.04.1 LTS
spark-2.3.2
Intel(R) Xeon(R) W-2145 CPU # 3.70GHz