I try to work with spark-sql but I had the following errors :
error: missing or invalid dependency detected while loading class file
'package.class'. Could not access term annotation in package
org.apache.spark, because it (or its dependencies) are missing. Check
your build definition for missing or conflicting dependencies. (Re-run
with -Ylog-classpath to see the problematic classpath.) A full
rebuild may help if 'package.class' was compiled against an
incompatible version of org.apache.spark. warning: Class
org.apache.spark.annotation.InterfaceStability not found - continuing
with a stub. error: missing or invalid dependency detected while
loading class file 'SparkSession.class'. Could not access term
annotation in package org.apache.spark, because it (or its
dependencies) are missing. Check your build definition for missing or
conflicting dependencies. (Re-run with -Ylog-classpath to see the
problematic classpath.) A full rebuild may help if
'SparkSession.class' was compiled against an incompatible version of
org.apache.spark.
My configuration :
Scala 2.11.8
Spark-core_2.11-2.1.0
Spark-sql_2.11-2.1.0
Note: I use SparkSession.
After dig into the error message, I know how to solve this kind of errors.
For example:
Error - Symbol 'term org.apache.spark.annotation' is missing... A full rebuild may help if 'SparkSession.class' was compiled against an incompatible version of org.apache.spark
Open SparkSession.class, search "import org.apache.spark.annotation.", you will find import org.apache.spark.annotation.{DeveloperApi, Experimental, InterfaceStability}. It's sure that these classes is missing in classpath. You'll need to find the artifact which conclude these classes.
So open https://search.maven.org and search with c:"DeveloperApi" AND g:"org.apache.spark", you will find the missing artifact is spark-tags as #Prakash answered.
In my situation, just add dependencies spark-catalyst and spark-tags in pom.xml works.
But it's weird that why maven not auto resolve transitive dependencies here?
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.0</version>
<scope>provided</scope>
</dependency>
If I use the above depencency, only spark-core_2.11-2.2.0.jar is in maven dependency; While if I change version to 2.1.0 or 2.3.0, all transitive dependencies will be there.
You need to include following artifacts to avoid the dependency issues.
spark-unsafe_2.11-2.1.1
spark-tags_2.11-2.1.1
Related
I've built a simple app with these dependencies:
[dependencies]
core-foundation = { version = "0.9", features = ["with-chrono"] }
With cargo 1.62.1 (a748cf5a3 2022-06-08). My platform is a Macbook Air M1 (the target might matter, not sure).
And I'm getting this error:
error: cyclic package dependency: package `chrono v0.4.21` depends on itself. Cycle:
package `chrono v0.4.21`
... which satisfies dependency `chrono = "^0.4"` of package `core-foundation v0.9.3`
... which satisfies dependency `core-foundation = "^0.9"` of package `iana-time-zone v0.1.42`
... which satisfies dependency `iana-time-zone = "^0.1.41"` of package `chrono v0.4.21`
If I clone core-foundation and build it myself using cargo build --features="with-chrono", I'm not getting any error.
Any idea how to investigate this issue?
Unfortunately, there isn't much you can do if the cyclic dependency occurs with crates you don't control. You could attempt to "fix" one of the crates by cloning and modifying it to avoid a problematic dependency and override your dependencies with the fixed version.
Your best bet is to notify the maintainers of the crates involved. If the issue only arose recently, you can try locking your dependencies to an earlier version as a workaround.
This particular issue seems to be resolved already. It likely happened when chrono added iana-time-zone as a dependency in version 0.4.21. However, the fix was for iana-time-zone to change its dependency from core-foundation to core-foundation-sys in version 0.1.45 to break the cycle.
I'm trying to include a specific version of a library in a Haskell project. The library is bed-and-breakfast (which is used for martix operations), but I need the specific version 0.4.3 which fixed a bug with the multiplication implementation.
So, my stack.yaml looks like this:
flags: {}
extra-package-dbs: []
packages:
- .
extra-deps:
- bed-and-breakfast-0.4
- base-4.6.0.1
resolver: lts-12.8
But I'm getting this error when building:
Error: While constructing the build plan, the following exceptions were encountered:
In the dependencies for bed-and-breakfast-0.4:
base-4.11.1.0 from stack configuration does not match >=4.5 && <4.7 (latest matching version
is 4.6.0.1)
needed due to realworldhaskell-0.1.0.0 -> bed-and-breakfast-0.4
Some different approaches to resolving this:
* Set 'allow-newer: true' to ignore all version constraints and build anyway.
* Consider trying 'stack solver', which uses the cabal-install solver to attempt to find some
working build configuration. This can be convenient when dealing with many complicated
constraint errors, but results may be unpredictable.
* Recommended action: try adding the following to your extra-deps
in C:\Users\info\Desktop\Projects\haskell\stack.yaml:
- base-4.6.0.1
I've done the recommended action but it didn't solve anything. I've tried using different resolvers to see if it's an issue with my GHCi version but nothings worked. What is the best way to interpret error messages like this and how should I proceed?
EDIT:
If I remove -base.4.6.0.1 and add allow-newer: true I get this:
WARNING: Ignoring out of range dependency (allow-newer enabled): base-4.11.1.0. bed-and-breakfast requires: >=4.5 && <4.7
bed-and-breakfast-0.4: configure
Progress 1/2
-- While building custom Setup.hs for package bed-and-breakfast-0.4 using:
C:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_Z6RU0evB_2.2.0.1_ghc-8.4.3.exe --builddir=.stack-work\dist\7d103d30 configure --with-ghc=C:\Users\info\AppData\Local\Programs\stack\x86_64-windows\ghc-8.4.3\bin\ghc.EXE --with-g
hc-pkg=C:\Users\info\AppData\Local\Programs\stack\x86_64-windows\ghc-8.4.3\bin\ghc-pkg.EXE --user --package-db=clear --package-db=global --package-db=C:\sr\snapshots\76fd1958\pkgdb --package-db=C:\Users\info\Desktop\Projects\haskell\
.stack-work\install\8c390635\pkgdb --libdir=C:\Users\info\Desktop\Projects\haskell\.stack-work\install\8c390635\lib --bindir=C:\Users\info\Desktop\Projects\haskell\.stack-work\install\8c390635\bin --datadir=C:\Users\info\Desktop\Proj
ects\haskell\.stack-work\install\8c390635\share --libexecdir=C:\Users\info\Desktop\Projects\haskell\.stack-work\install\8c390635\libexec --sysconfdir=C:\Users\info\Desktop\Projects\haskell\.stack-work\install\8c390635\etc --docdir=C:
\Users\info\Desktop\Projects\haskell\.stack-work\install\8c390635\doc\bed-and-breakfast-0.4 --htmldir=C:\Users\info\Desktop\Projects\haskell\.stack-work\install\8c390635\doc\bed-and-breakfast-0.4 --haddockdir=C:\Users\info\Desktop\Pr
ojects\haskell\.stack-work\install\8c390635\doc\bed-and-breakfast-0.4 --dependency=array=array-0.5.2.0 --dependency=base=base-4.11.1.0 --dependency=binary=binary-0.8.5.1 --dependency=deepseq=deepseq-1.4.3.0 --dependency=template-hask
ell=template-haskell-2.13.0.0 --extra-include-dirs=C:\Users\info\AppData\Local\Programs\stack\x86_64-windows\msys2-20150512\mingw64\include --extra-lib-dirs=C:\Users\info\AppData\Local\Programs\stack\x86_64-windows\msys2-20150512\min
gw64\bin --extra-lib-dirs=C:\Users\info\AppData\Local\Programs\stack\x86_64-windows\msys2-20150512\mingw64\lib --exact-configuration
Process exited with code: ExitFailure 1
Logs have been written to: C:\Users\info\Desktop\Projects\haskell\.stack-work\logs\bed-and-breakfast-0.4.log
Configuring bed-and-breakfast-0.4...
Cabal-simple_Z6RU0evB_2.2.0.1_ghc-8.4.3.exe: The package has an impossible
version range for a dependency on an internal library: bed-and-breakfast
==0.3.2. This version range does not include the current package, and must be
removed as the current package's library will always be used.
EDIT 2:
Ok, so I'm guessing that the bed-and-breakfast library needs base 4.11.1.0 which is included in GHCi 6.10.2 (according to https://wiki.haskell.org/Base_package) so I need a resolver which matches that GHCi version. Where can I find out what resolver version that is?
Here is the constraint on the latest available bed-and-breakfast package: base (>=4.5 && <4.7), which means it will likely not even compile with GHC version higher then 7.6. Considering that there is even no LTS snapshot prior to GHC 7.8, you are out of luck with that package.
To say it in another words, the package is outdated and your choices are:
submit an issue and hope maintainer will do something about it
try to make it work with newer ghc yourself
Use a different package
When I include the following in my build.sbt (using Scala version 2.12.1)
"org.reactivemongo" %% "reactivemongo" % "0.12.1",
"org.reactivemongo" %% "reactivemongo-play-json" % "0.12.1",
I get the error...
Error:scalac: missing or invalid dependency detected while loading class file 'LowerImplicitBSONHandlers.class'.
Could not access type Writes in package play.api.libs.json,
because it (or its dependencies) are missing.
Check your build definition for missing or conflicting dependencies
(Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'LowerImplicitBSONHandlers.class' was compiled against an incompatible version of play.api.libs.json.
Can reactivemongo and reactivemongo-play-json live together?
Thanks,
Nathaniel
Seems you need to include play-json as a dependency if play isn't already included in your dependencies. A bit strange, since I thought the whole idea of reactivemongo-play-json was to get the json support without requiring play. Regardless, compilation now works after including the following dependency:
"com.typesafe.play" %% "play-json" % "2.6.0-M3"
I've been iterating on features in my first Gradle plugin. I determined early on that I needed commons-io, so I added a dependency on commons-io 2.4, being the latest version.
This has been going well for a while, with the build working from the command line, and no errors in Eclipse.
I just started trying to integrate some code that uses "FileUtils.write(File,String)". I didn't need that method before. I got everything un-redded in Eclipse, and then I tried a command line build.
This failed with errors like the following:
... error: cannot find symbol
FileUtils.write(serviceLoaderFile,
^
symbol: method write(File,String)
location: class FileUtils
This confused me. I went to the failing lines in Eclipse, and no issues were indicated. I navigated into the "write()" method, and it looked fine to me. I then ran my command-line build with "--debug" to get some clues.
When I found the "javac" line, I found that "$GRADLE_HOME\lib\commons-io-1.4.jar" (where "GRADLE_HOME" is just my Gradle 2.3 distribution) was in the classpath BEFORE my dependency jar. I then inspected the code in the 1.4 jar, and I determined that the "FileUtils" class in that version didn't have a "write" method.
What am I supposed to do about this?
Update:
I suppose it's likely my "dependencies" block would be useful, which is this:
dependencies {
compile ("org.codehaus.groovy:groovy-all:2.3.9")
compile gradleApi()
compile "org.opendaylight.yangtools:yang-parser-impl:0.7.0-SNAPSHOT"
compile "org.opendaylight.yangtools:binding-java-api-generator:0.7.0-SNAPSHOT"
compile "org.opendaylight.yangtools:binding-generator-api:0.7.0-SNAPSHOT"
compile "org.opendaylight.yangtools:binding-generator-impl:0.7.0-SNAPSHOT"
compile "org.opendaylight.controller:yang-jmx-generator:0.3.0-SNAPSHOT"
compile "commons-io:commons-io:2.4"
testCompile("org.spockframework:spock-core:1.0-groovy-2.3") {
exclude group: "org.codehaus.groovy"
}
I tried commenting out the "gradleApi" reference, and that had no effect. I also tried adding an "exclude" for commons-io associated with the "groovy-all" reference, but that also didn't appear to make any difference.
}
You probably added gradleApi() under dependencies {} block - see the docs. The problem is that it ships all dependencies gradle requires - including commons-io in version 1.4 - see the extract below:
[opal#opal-mac-2]~/.gvm/gradle/current/lib % pwd
/Users/opal/.gvm/gradle/current/lib
[opal#opal-mac-2]~/.gvm/gradle/current/lib % ll commons-io-1.4.jar
-rw-rw-r-- 1 opal staff 109043 23 gru 13:17 commons-io-1.4.jar
[opal#opal-mac-2]~/.gvm/gradle/current/lib %
You probably added version 2.4 separately and that's why conflict occurred. You can also run
gradle dependencies
to view the full dependency tree and verify the problem.
There's no possibility to exclude a transitive dependency from gradleApi().
The solution to this required adding the following block to the "sourceSets" block:
main {
compileClasspath = configurations.compile.minus files("$gradle.gradleHomeDir/lib/commons-io-1.4.jar")
}
This is pretty simple, but I wish there was a cleaner solution for this. I'm not sure what that would look like.
https://travis-ci.org/Fresheyeball/fbatch
Preprocessing library monad-parallel-0.5...
Control/Monad/Parallel.hs:67:8:
Could not find module `Control.Monad.Identity'
It is a member of the hidden package `mtl-2.2.1'.
Perhaps you need to add `mtl' to the build-depends in your .cabal file.
Use -v to see a list of the files searched for.
cabal: Error: some packages failed to install:
monad-parallel-0.5 failed during the building phase. The exception was:
ExitFailure 1
I am new to haskell, and just can't seem to get past this. I've tinkered with my .cabal file all I can, and the sub dependency fails on Travis every time.
Please help.
If a build fails while resolving a dependency, it's worth having a look a what version of that package gets installed. Normally, cabal should default to the most recent possible, which if well-maintained should install fine. Older versions do of course often mean trouble, if some dependency of theirs has now a newer, incompatible version installed.
In your case, cabal tried to install the somewhat ancient monad-parallel-0.5, which imports a module that doesn't exist in transformers (namely Control.Monad.Identity). The more recent version 0.7.1.2 does not try this and thus builds successfully. You can force such a recent install by giving a lower bound to the dependency in your .cabal file, in this case monad-parallel >= 0.7.