Trying to build the image from the provided download. Used "Source Code" package option download, not Hadoop versions. Following spark.apache.org/docs/2.3.0/running-on-kubernetes.html#docker-images first step to build a docker image. Have Docker CE with K8s running on Mac.
Using Terminal: Ran in spark folder:
./bin/docker-image-tool.sh -r cloud.docker.com/xxxx/spark build
All runs until here:
Step 5/14 : COPY ${spark_jars} /opt/spark/jars
COPY failed: stat /var/lib/docker/tmp/docker-builder840552377/assembly/target/scala-2.12/jars: no such file or directory
Please assist a newbie! Not sure where this folder structure is or whats up.
Please don't assume I did something correct....I could have missed an obvious install before running this. Help is appreciated after you laugh. :-)
Thanks
Make sure you've cloned the right spark project and then Build spark with this command
./build/mvn -Pkubernetes -DskipTests clean package
If you just need to solve your current error, use this command (-pl is used to build a specified module where spark-assembly_2.11 is the artifactId as defined in assembly/pom.xml file.)
build/mvn -pl :spark-assembly_2.11 clean install
The scala jar will be available in the Assembly module folder.
Related
I am trying to generate code coverage report using LCOV in Ubuntu platform. I referred
https://github.com/QianYizhou/gtest-cmake-gcov-example
It is working.
I am giving cmake --build ../application/build --target install in my shell script.
After script execution, I can see that the cmakefiles are generated in the build folder.
cd build && make test
cd build && make coverage_TEST_NAME //To check the coverage
I did execute above in my build folder and I generated report.
My problem is, I use this in gitlab pipeline. There is no build folder I can see. So I don't know how to do make coverage_TEST_NAME in my yml file
Generate code generation output in gitlab pipeline.
Problem resolved. Just like in my Ubuntu virtual machine, the application folder was there in my docker image.
I just added a ../application/build command in my yml script, it navigated there.
I have a leshan server jar file (to which I have made some changes) obtained by running the maven clean install. I specify that I work in linux and I put this jar file inside a "leshan_docker" folder contained in the desktop. within the same folder there is also a dockerfile to build the server image, and it is written as follows:
FROM openjdk:8-jre-alpine
COPY /Desktop/leshan_docker/leshan-server-demo-*.jar /Desktop/leshan_docker/
CMD ["java", "-jar", "/leshan-server-demo-2.0.0-SNAPSHOT.jar"]
but when I go to build through this command:
sudo docker build -f Dockerfile3 -t leshan-server3 .
It reports me the following error:
Sending build context to Docker daemon 12MB
Step 1/3 : FROM openjdk:8-jre-alpine
---> f7a292bbb70c
Step 2/3 : COPY /Desktop/leshan_docker/leshan-server-demo-*.jar /Desktop/leshan_docker/
COPY failed: no source files were specified
How can I go about solving the problem? Thanks in advance for your answers.
Your source path with the COPY command should be relative to the build context. Your build context is in the folder you're running sudo docker build in since the final argument you gave was .. I highly recommend taking a look at the docs.
The destination path for the COPY command should be relative to the path in your container. What may work now is to move your .jar to the root directory and run it from there.
So if your jar files are in the same directory you're running the command in, change it to:
COPY leshan-server-demo-*.jar /
It would be better practice to actually create a new directory in the container to hold your .jar file to keep your work more organized.
click here to see the image
i could not create maven project in eclipse IDE.
i am getting the error in the image and as well as mentioned below
Could not calculate build plan: Plugin
org.apache.maven.plugins:maven-resources-plugin:2.6 or one of its
dependencies could not be resolved: Failed to read artifact descriptor
for org.apache.maven.plugins:maven-resources-plugin:jar:2.6 Plugin
org.apache.maven.plugins:maven-resources-plugin:2.6 or one of its
dependencies could not be resolved: Failed to read artifact descriptor
for org.apache.maven.plugins:maven-resources-plugin:jar:2.6**
i have tried following:
deleted the folders and files under .m2/repositary and it did not work
windows->preferences->maven->user settings->and pointed the settings.xml to the downloaded apache maven 3.5.2 and it also did not work.
created the settings.xml under .m2/repositary ,it also did not work.
environment variable also set for maven(maven_home and m2_home) and java_home.
i have googled and tried all these but could not resolve this issue.
any help would be appreciated .
Thanks in advance.
First, make sure to use .m2/repository, not .m2/repositary
Check that your settings.xml is correctly set, with at least:
<mirror>
<id>Central</id>
<url>http://repo1.maven.org/maven2</url>
<mirrorOf>central</mirrorOf>
</mirror>
Then check in a CMD (meaning outside) Eclipse, if:
M2_HOME and JAVA_HOME are also correctly set there
java -version does return the expected version
mvn -version does return the expected version
you can download the artifact:
For the last point, you actually can execute a mvn command, anywhere you want (no need for a pom.xml)
cd c:\
mvn org.apache.maven.plugins:maven-dependency-plugin:3.0.2:get -Dartifact=org.apache.maven.plugins:maven-resources-plugin:2.6 -U
I am following instructions from here:
https://www.datacamp.com/community/tutorials/apache-spark-python#gs.WEktovg
I downloaded and prebuilt version of Spark , untarred it and mv it to /usr/local/spark.
According to this, this is all I should have to do.
Unfortunately, I can run the interactive shell as it cant find the file.
When i run :
./bin/pyspark
I get
-bash: ./bin/pyspark: No such file or directory.
I also notice that installing it this way does not add it to the bin directory.
Is this tutorial wrong or am I missing a trick?
You need to change your working directory to /usr/local/spark. Then this command will work.
And also, when you untar it, it usually will not add it to bin folder. You need to add it manually by adding the path to environment variables.
Update your working Directory to /usr/local/spark and execute the command. Hopefully this will fix the issue.
I found https://github.com/mozilla/gecko-dev/tree/master/b2g Mozilla repositories where have the moz.build file, can anybody help, how I can build this plugin. What are tools I need use for build project with use moz.build ?
thank you
If you are trying to build B2G (Firefox OS), then you should follow the procedure documented here. But I have to advise you that you're using the wrong repository if that's your objective, since the correct one for B2G should be this.
In any case, you would need to install the build prerequisites for Linux, as described here, by using the following command:
wget -O bootstrap.py https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py && python bootstrap.py
Now, if you really wanted to just build the b2g folder in the repository you linked, once you've downloaded and installed the prerequisites, simply issue the following command from the root of gecko-dev:
./mach build b2g
This will invoke mach, the build system Mozilla uses, to build the code in the b2g directory.