Connecting Blob to Spark failing - apache-spark

Question

I am trying to connect spark coming in Hortonworks 2.4 distribution to Azure Blob storage and I was getting error "wasb filesystem not recognized " . I checked and many suggested downloading azure-sdk-for-java and creating package.
I am trying to create package using mvn but while running both mvn test and mvn package ; the process is getting hanged while testing. There is no error given , it just failed to return anything. I had changed testconfiguration.xml to reflect my blob account name. Below is the log files I am getting.
Is there something else that needs to be done to get jars ?
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Microsoft Azure Storage Client SDK 4.3.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) # azure-storage ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/adminhorton/kspark/azure-storage-java-master/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) # azure-storage ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.5:testResources (default-testResources) # azure-storage ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) # azure-storage ---
[INFO] Compiling 52 source files to /home/adminhorton/kspark/azure-storage-java-master/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.13:test (default-test) # azure-storage ---
[INFO] Surefire report directory: /home/adminhorton/kspark/azure-storage-java-master/target/surefire-reports
T E S T S
parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
Running com.microsoft.azure.storage.StorageAccountTests
Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.057 sec
Running com.microsoft.azure.storage.StorageUriTests
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.071 sec
com.microsoft.azure.storage.StorageAccountTests.txt
com.microsoft.azure.storage.StorageUriTests.txt
com.microsoft.azure.storage.StorageAccountTests.txt
com.microsoft.azure.storage.StorageUriTests.txt

Answer 1

According to your description, I am not sure you have configured your Storage Configuration rightly in your project. So I recommend you can follow these steps to check your configurations:
1.Create your Azure storage Account and set it in core-site.xml as this format:
<property>
<name>fs.azure.account.key.youraccount.blob.core.windows.net</name>
<value>YOUR ACCESS KEY</value>
</property>
2.Restart the HDP service and use the Hadoop fs –ls wasb://**.blob.core.windows.net/ to list the files in the container.
I strongly recommend you can refer to this blog how-to-configure-hortonworks-hdp-to-access-azure-windows-storage and official document.
Meanwhile , #Yuval provided the great sample about how to connect to Azure blob with Java SDK.