How can I deploy spark artifacts to a custom maven repository? - apache-spark

I have a fork of apache spark on github that I build from.
I have a maven repository that our company uses to manage dependencies.
Suppose the url of this maven repository is https://xyz.abc.com/maven.
If I run a command like
./build/mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.5 -DskipTests install
Then all the jars get deployed to my local maven repository, but what I would really like is for them to be deployed to the repository mentioned above.
How can I do that? Do I need to modify certain files or set environment variables? Do I need to add a credential file somewhere?
I also looked into doing it with sbt, but I could not figure out how to do it with that either.

You want to publish artifact build by maven to remote repository (https://xyz.abc.com/maven).
Take a look at this guide for required configuration.

Related

Jenkin build looking for both repositories for given jar and failing when not present in it

I have a jar xbean-asm9-shaded-4.20.jar present in Nexus repository2 and I have mentioned that repository under in pom.xml. There is one more Nexus repository repository1 mentioned under for other artifacts.
While running Jenkins build it also tries to download xbean-asm9-shaded-4.20.jar from repository1 and fails even though repository2 is mentioned before repository1 in pom.xml.
How can I make it look only under repository2 for xbean-asm9-shaded-4.20.jar

How do you add JOOQ's 3.17 snaphot build to maven?

Does JOOQ push the latest non-stable builds to maven central? Or is there some snapshot repo I need to add to pom.xml?
Does JOOQ push the latest non-stable builds to maven central?
Snapshot builds are offered only to paying customers, available from here: https://www.jooq.org/download/versions
Or is there some snapshot repo I need to add to pom.xml?
The above website offers ZIP file downloads. The ZIP file contains two scripts:
maven-install.sh and maven-install.bat to install the snapshot artifacts locally, to your local repository (e.g. ~/.m2)
maven-deploy.sh and maven-deploy.bat to deploy the snapshot artifacts to your artifact repository
There are other options, e.g. using the maven-install-plugin in your build, of course, or build and install the snapshots directly from source code

How to create a build pipeline for multi-module maven in Azure Devops

I have a multi-module maven project in azure repo:
Parent
----App
--POM.xml
----Core
--POM.xml
----API
--POM.xml
----ParentPOM.xml
When I tried to build App project in Azure, I am getting the following error: [ERROR] Failed to execute goal on the project: Could not resolve dependencies for project com.core:jar:1.0: Could not find artifact com.core:jar:1.0 in central (https://repo.maven.apache.org/maven2).
So I tried to create multiple tasks in the same azure build to package for each module (App, Core, API), which also gives the same result. Could someone please help me on how to build the multi-module project and get the App jar file.
The way I've done it:
with this command: install -pl (module name) -am
where the module name should be one of the submodules. If you want to build all of them then only go with install
I have found the solution for the multi-module maven dependency issue, First We need to build the parent pom.xml, then build the submodules as per hierarchical order. In between each module, add a task to copy and move the jar files to the staging directory. Here shared the image of my build steps. Please take a look and comment on any questions.
Please look into the DevOps configuration for reference

Publish Gitlab artifacts to Artifactory

I want to publish my Gitlab project's(not a maven project) artifact to JFrog Artifactory. The artifact size is 4.2 GB.
I searched for this but mostly got links to publish Gitlab Maven project to Artifactory. My project is not a maven project.
I have a requirement to keep all source code in Gitlab and artifacts(.war, .tar.gz) in Artifactory.
How do I achieve this?
It sounds like you're looking for Git LFS. This is an extention to Git that allows your Gitlab repository to track artifacts without actually storing them in the repository, instead using some external filestore or artifact management server.
Artifactory supports Git LFS repositories, and you can find the documentation for setting it up here.

How host local repository in pom.xml

When ever i tried to Goals >> Compile pom.xml, I am getting the following error.
An internal error was encountered invoking the maven goal: Please check the exception details.
org.apache.maven.artifact.resolver.MultipleArtifactsNotFoundException: Missing:
1) org.apache.httpcomponents:httpcore:jar:4.1.2
Try downloading the file manually from the project website.
Then, install it using the command:
mvn install:install-file -DgroupId=org.apache.httpcomponents -DartifactId=httpcore -Dversion=4.1.2 -Dpackaging=jar -Dfile=/path/to/file
Alternatively, if you host your own repository you can deploy the file there:
mvn deploy:deploy-file -DgroupId=org.apache.httpcomponents -DartifactId=httpcore -Dversion=4.1.2 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
Path to dependency:
1) org.apache.httpcomponents:httpcore:jar:4.1.2
1 required artifacts are missing.
As per above error it suggests, Alternatively, if you host your own repository you can deploy the file there.
how can i provide a local repository to pom.xml.
This article explains what you require. Quoting from it:
A maven repository is just a directory somewhere, and that’s about it. In order to use it there are very few steps you need to do.
Choose a directory
Make sure its accessible via some network protocol to everyone who needs it (i.e. serve it via http or ftp or something.
Put the libraries you need in the directory, in a specific format.
Set up your maven builds to read from the repository
(optional) set up your maven builds to deploy your repository
To put existing libraries into the repository, use the maven deploy command.
To set up your build to read from the repository add the following xml to your pom.xml file:
servername-download
description
https://host/path
A detailed article for complete understanding of Maven repositories
Are you sure your Maven setup works? The artifact you're looking for is in the public Maven repository:
http://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.1.2/
It should be found just like any other publicly available artifact. Please check the spelling of the artifact and group in your pom file to make sure that you don't have a typo.
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
<version>4.1.2</version>
</dependency>
What does Maven print before it fails? It usually says where it's checking for the artifact - does the URL I posted above show up?
Do you see similar issues with other artifacts?
You shouldn't have to host this artifact yourself since it's publicly available. It looks like you're running Maven from an IDE - can you try to run it from command line to see if you get the same error there?
To host local repository in pom.xml we have to set up maven properly.
we need to modify settings.xml and provide the local repository path in maven packages and then used it with IDE. So now on words if JAR is not available in my local repository then and then only maven tries to search in its maven repository.
please follow below blog to configure maven properly.
http://chiragsdiary.blogspot.in/2013/02/cxf-webservice-example-step-by-step.html

Resources