I am running a job from a jar file in Azure Databricks. This jar has a dependency on azure-storage-file-share. Previously, there wasn't an issue installing this dependency using Maven within the Databricks UI. Now, I get failure with this error message:
Run result unavailable: job failed with error message Library installation failed for library due to user error for maven { coordinates: "com.azure:azure-storage-file-share:12.16.2" } Error messages: Library installation attempted on the driver node of cluster 0131-144423-r1g81134 and failed. Please refer to the following error message to fix the library or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File file:/local_disk0/tmp/clusterWideResolutionDir/maven/ivy/jars/io.netty_netty-transport-native-kqueue-4.1.86.Final.jar does not exist
To try to work around this, I manually installed this netty library (and several others) as well. I can see in the logs that it was able to download the jar successfully:
downloading https://maven-central.storage-download.googleapis.com/maven2/io/netty/netty-transport-native-kqueue/4.1.86.Final/netty-transport-native-kqueue-4.1.86.Final-osx-x86_64.jar ... [SUCCESSFUL ] io.netty#netty-transport-native-kqueue;4.1.86.Final!netty-transport-native-kqueue.jar (202ms)
However, it still fails. This is error message is in the same log after the one above:
23/01/31 14:50:09 WARN LibraryState: [Thread 137] Failed to install library maven;azure-storage-file-share;com.azure;12.16.2;; java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File file:/local_disk0/tmp/clusterWideResolutionDir/maven/ivy/jars/io.netty_netty-transport-native-kqueue-4.1.86.Final.jar does not exist
when running:
mvn com.microsoft.azure:azure-webapp-maven-plugin:1.14.0:config
I end up with
[ERROR] Failed to execute goal com.microsoft.azure:azure-webapp-maven-plugin:1.14.0:config (default-cli) on project storingen-api: Max retries 0 times exceeded. Error Details: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.microsoft.azure:azure-webapp-maven-plugin:1.14.0:config (default-cli) on project storingen-api: Max retries 0 times exceeded. Error Details: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Clearly this is a certificate problem that cannot be found. I have however no clue what certificate is needed nor in which keystore it needs to be. I suppose in the cacerts of my java distribution but it's not clear in any way.
Does anyone know which cert and where to put it ?
I just running into the same problem following Azure Learn Java Azure
Basically, this error ocurred because Java is trying to connect with your Azure Subscription at Azure Console. I saw the origin of this problem executing Maven command with -X. The log trace can be visualized over here, like this:
mvn com.microsoft.azure:azure-webapp-maven-plugin:1.12.0:config -X
What I did to solve:
I tried inserting Azure certifications inside Java cacerts - following the explanation here. But it didn't change or solved the problem.
I searched for a new one version of azure-webapp-maven-plugin. Nowadays, the version is 2.3.0. You can check at Maven repository the last version. It fixed my problem and no more throws the error caused by Java.
before: mvn com.microsoft.azure:azure-webapp-maven-plugin:1.12.0:config
after (I think so, that version fixes the problem): mvn com.microsoft.azure:azure-webapp-maven-plugin:2.3.0:config
Thanks
I'm using Ops center v5.2.4 to install DataStax Enterprise v4.8.1 in a local cluster and following guide:
[http://docs.datastax.com/en/opscenter/5.2/opsc/online_help/opscCreateClusterLocal.html][1],
but it keeps failing due to error(/var/log/datastar-agent/agent.log):
INFO [Thread-22] 2016-04-29 09:19:18,777 Checking for already installed packages
INFO [Thread-22] 2016-04-29 09:19:18,778 Running apt-get install dry-run
INFO [Thread-22] 2016-04-29 09:19:19,347 Running sudo command.
ERROR [Thread-22] 2016-04-29 09:19:19,381 Installation failed: #<ExceptionInfo clojure.lang.ExceptionInfo: throw+: {:type :installation-error, :message "Could not find a matching versi$
clojure.lang.ExceptionInfo: throw+: {:type :installation-error, :message "Could not find a matching version for package dse-libpig"}
deb.clj:138 opsagent.installation.deb/make-versioned-package
core.clj:2487 clojure.core/map[fn]
LazySeq.java:42 clojure.lang.LazySeq.sval
LazySeq.java:67 clojure.lang.LazySeq.seq
RT.java:484 clojure.lang.RT.seq
core.clj:133 clojure.core/seq
core.clj:687 clojure.core/concat[fn]
LazySeq.java:42 clojure.lang.LazySeq.sval
LazySeq.java:60 clojure.lang.LazySeq.seq
RT.java:484 clojure.lang.RT.seq
core.clj:133 clojure.core/seq
core.clj:2605 clojure.core/drop-while[fn]
core.clj:2609 clojure.core/drop-while[fn]
LazySeq.java:42 clojure.lang.LazySeq.sval
LazySeq.java:60 clojure.lang.LazySeq.seq
RT.java:484 clojure.lang.RT.seq
core.clj:133 clojure.core/seq
core.clj:617 clojure.core/apply
shell.clj:50 clojure.java.shell/parse-args
shell.clj:112 clojure.java.shell/sh
RestFn.java:137 clojure.lang.RestFn.applyTo
core.clj:617 clojure.core/apply
proc.clj:158 opsagent.proc/sudo
RestFn.java:423 clojure.lang.RestFn.invoke
deb.clj:157 opsagent.installation.deb/dry-run
Var.java:419 clojure.lang.Var.invoke
AFn.java:163 clojure.lang.AFn.applyToHelper
Var.java:532 clojure.lang.Var.applyTo
core.clj:617 clojure.core/apply
install.clj:20 opsagent.installation.install/package-specific
RestFn.java:439 clojure.lang.RestFn.invoke
install.clj:76 opsagent.installation.install/run-installation[fn]
AFn.java:24 clojure.lang.AFn.run
Thread.java:745 java.lang.Thread.run
I'm sure the credential is right and ssh is enabled.
Can anyone help to get me out?
I have a fresh installation of Groovy 2.1.4 and I'd like to create a script that uses HTTP builder.
I've added the following line at the top of the script:
#Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.6')
When I run the script in GroovyConsole, I get the following error:
1 compilation error:
Exception thrown
VI 01, 2013 12:15:39 ODP. org.codehaus.groovy.runtime.StackTraceUtils sanitize
WARNING: Sanitizing stacktrace:
java.lang.RuntimeException: Error grabbing Grapes -- [download failed: commons-logging#commons-logging;1.1.1!commons-logging.jar]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
... (aso) ...
java.lang.RuntimeException: Error grabbing Grapes -- [download failed: commons-logging#commons-logging;1.1.1!commons-logging.jar]
When I run the following command on the Windows command line:
grape -V resolve org.codehaus.groovy.modules.http-builder http-builder 0.6
I get the same error, i.e.:
:: problems summary ::
:::: WARNINGS
[NOT FOUND ] commons-logging#commons-logging;1.1.1!commons-logging.jar (0ms)
==== localm2: tried
file:C:\Documents and Settings\Administrator/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Error in resolve:
Error grabbing Grapes -- [download failed: commons-logging#commons-logging;1.1.1!commons-logging.jar]
When I try to resolve commons-logging directly, via the following command:
grape -V resolve commons-logging commons-logging 1.1.1
I get the same error, i.e. not found.
When I instead try to download the latest version, i.e. 1.1.3, via the appropriate command, it works. But even after that, resolving the http-client still fails.
Where's the problem?
Try deleting ~/.m2 directory and also ~/.groovy/grapes directories.
It worked for me.
It looks like your Grape settings are set only to look in your local .m2 directory for the library. Check your grapeConfig.xml configuration file. (If you don't have one, you can create it--it should go in the same directory where your groovysh.history and grapes cache directory are created by Groovy.) You can copy the example file shown on the Groovy Grape reference page.
If that doesn't help, I would try deleting your grapes cache directory and try it again.
My problem was, groovy (v2.4.8) was looking in the maven repository fist (~/.m2/repository/) and finding the pom file but not finding the associated artifact/JAR. Instead of just moving on to the next resolver, which would have succeeded, it just gives up. The workaround would be to remove the pom file, the specific directory cache, or just temporarily rename the repository and run groovy again as other suggested. Or you could try to manually add it to the repository. But these are just temporary and you'll likely run into the issue again if you clear your groovy cache or with another dependency.
To troubleshoot this issue you can turn on verbose logging and try to manually install the dependency. So if your error is something like:
java.lang.RuntimeException: Error grabbing Grapes -- [download failed: commons-logging#commons-logging;1.1.1!commons-logging.jar, download failed: commons-codec#commons-codec;1.6!commons-codec.jar, download failed: commons-lang#commons-lang;2.4!commons-lang.jar]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
Which means your dependency is commons-logging-1.1.1.jar. You can run:
grape -V install commons-logging commons-logging 1.1.1
What ended up solving the problem for me was overriding the default configuration and setting usepoms="false" in the localm2 resolver. It works in my case because I have the pom but not the JAR, so since Ivy isn't considering the POM now and the JAR was never there to begin with, it goes onto the next resolver which does find it. So in summary:
Create this file: ~/.groovy/grapeConfig.xml
With these contents:
<ivysettings>
<settings defaultResolver="downloadGrapes"/>
<resolvers>
<chain name="downloadGrapes" returnFirst="true">
<filesystem name="cachedGrapes">
<ivy pattern="${user.home}/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml"/>
<artifact pattern="${user.home}/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision](-[classifier]).[ext]"/>
</filesystem>
<ibiblio name="localm2" root="file:${user.home}/.m2test/repository/" checkmodified="true" changingPattern=".*" changingMatcher="regexp" m2compatible="true" usepoms="false"/>
<ibiblio name="jcenter" root="https://jcenter.bintray.com/" m2compatible="true"/>
<ibiblio name="ibiblio" m2compatible="true"/>
</chain>
</resolvers>
</ivysettings>
I was tempted not to use the maven 2 cache at all, but if I removed the line, I got errors about not being able to find "localm2". Although updating it to point to fictitious directory worked.
Deleting .m2 and grapes did not resolve the issue for me as when they were repopulated, the jar was not being downloaded. In my case, I was missing xml-apis-1.3.04.jar
I eventually resolved the issue by manually downloading the jar file and copying it into ~/.m2/repository/xml-apis/xml-apis/<version>
Hopefully this is useful if simply deleting the directories isn't working.
this might help someone down the line as i faced similar issue in windows
java.lang.RuntimeException: Error grabbing Grapes -- [download failed: commons-collections#commons-collections;3.2.2!commons-collections.jar]
the required jar above was not being downloaded in C:/Users/%USER%/.groovy/grapes/common-collections/jars/ folder.
so manually downloaded the required version from https://jar-download.com/
and added in the corresponding folder
Not quite the same case as the question, adding it in case someone finds himself in a similar case with me where I did not have any direct access on the machine to check the grapeConfig.xml configuration file, as suggested by the accepted answer. What worked for me was to configure a grab resolver.
I did something similar with what is stated in the documentation and it worked (documentation link):
#GrabResolver(name='restlet', root='http://maven.restlet.org/')
#Grab(group='org.restlet', module='org.restlet', version='1.1.6')
I solved a similar issue by using a newer version of commons-codec:commons-codec in groovy script using the #Grab annotation:
#Grapes([
#Grab('org.slf4j:slf4j-simple:1.7.25'),
#Grab('commons-codec:commons-codec:1.14'),
#Grab('io.github.http-builder-ng:http-builder-ng-apache:1.0.4')
])
you need create ~/.groovy/grapeConfig.xml
coments
<ivysettings>
<settings defaultResolver="downloadGrapes"/>
<resolvers>
<chain name="downloadGrapes" returnFirst="true">
<filesystem name="cachedGrapes">
<ivy pattern="${user.home}/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml"/>
<artifact pattern="${user.home}/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision](-[classifier]).[ext]"/>
</filesystem>
<ibiblio name="localm2" root="file:${user.home}/.m2test/repository/" checkmodified="true" changingPattern=".*" changingMatcher="regexp" m2compatible="true" usepoms="false"/>
<ibiblio name="jcenter" root="https://jcenter.bintray.com/" m2compatible="true"/>
<ibiblio name="ibiblio" m2compatible="true"/>
</chain>
</resolvers>
</ivysettings>
I had a similar error when using Groovy + Java 7.
The error from groovy was:
General error during conversion: Error grabbing Grapes -- [unresolved dependency: com.microsoft.sqlserver#mssql-jdbc;6.4.0.jre7: not found]
When trying the
grape -V resolve com.microsoft.sqlserver mssql-jdbc 6.4.0.jre7
command i had these errors:
java.net.SocketException: Connection reset
javax.net.ssl.SSLException: Received fatal alert: protocol_version
The way to fix this was adding the -Dhttps.protocols=TLSv1.2 parameter:
grape -Dhttps.protocols=TLSv1.2 -V resolve com.microsoft.sqlserver mssql-jdbc 6.4.0.jre7
Then the package is downloaded and i can use it from Groovy
Using the new DSE 3.0 Visual Cluster Provisioning tool I receive an error during node initialization - Could not find a matching version for package dse-libpig
/var/lib/opscenter-agent/agent.log contains:
INFO [Thread-9] 2013-02-26 15:30:41,929 Running apt-get install dry-run ERROR [Thread-9] 2013-02-26 15:30:42,966 Installation failed:
#<RuntimeException java.lang.RuntimeException: slingshot.ExceptionInfo: throw+: {:type :installation-error, :message Could not find a matching version for package dse-libpig}> java.lang.RuntimeException: slingshot.ExceptionInfo: throw+: {:type :installation-error, :message Could not find a matching version for package dse-libpig}
LazySeq.java:47 clojure.lang.LazySeq.sval
LazySeq.java:63 clojure.lang.LazySeq.seq
ChunkedCons.java:59 clojure.lang.ChunkedCons.chunkedNext
ChunkedCons.java:43 clojure.lang.ChunkedCons.next
RT.java:560 clojure.lang.RT.next
core.clj:61 clojure.core/next
core.clj:3399 clojure.core/nthnext
core_print.clj:55 clojure.core/print-sequential
core_print.clj:138 clojure.core/fn
MultiFn.java:167 clojure.lang.MultiFn.invoke
core.clj:2812 clojure.core/pr-on
core.clj:2824 clojure.core/pr
AFn.java:161 clojure.lang.AFn.applyToHelper
RestFn.java:132 clojure.lang.RestFn.applyTo
core.clj:540 clojure.core/apply
core.clj:2830 clojure.core/pr
RestFn.java:139 clojure.lang.RestFn.applyTo
core.clj:540 clojure.core/apply
core.clj:2863 clojure.core/print
RestFn.java:137 clojure.lang.RestFn.applyTo
core.clj:540 clojure.core/apply
core.clj:3716 clojure.core/print-str
RestFn.java:436 clojure.lang.RestFn.invoke
proc.clj:136 opsagent.proc/sudo
RestFn.java:423 clojure.lang.RestFn.invoke
deb.clj:157 opsagent.installation.deb/dry-run
Var.java:369 clojure.lang.Var.invoke
AFn.java:163 clojure.lang.AFn.applyToHelper
Var.java:482 clojure.lang.Var.applyTo
core.clj:540 clojure.core/apply
install.clj:18 opsagent.installation.install/package-specific
RestFn.java:439 clojure.lang.RestFn.invoke
install.clj:50 opsagent.installation.install/run-installation[fn]
AFn.java:24 clojure.lang.AFn.run
Thread.java:662 java.lang.Thread.run Caused by: slingshot.ExceptionInfo: throw+: {:type :installation-error, :message Could not find a matching version for package dse-libpig}
deb.clj:138 opsagent.installation.deb/make-versioned-package
core.clj:2096 clojure.core/map[fn]
LazySeq.java:42 clojure.lang.LazySeq.sval
INFO [Thread-9] 2013-02-26 15:30:42,967 Request 737078db-3e68-46e9-bf95-88c9921e0451 finished with state :error : Could not find a matching version for package dse-libpig
This is a bug in how OpsCenter specifies version numbers for the packages. It is using 3.0.0 when the packages are actually 3.0.
You can fix it temporarily by editing '/etc/opscenter/definitions/packages-1.json' and replacing '3.0.0' with '3.0' in the list of versions for DSE.
OpsCenter 3.0.1 is now available, and fixes this.