I am using a sigmajs library for creating node based graph visualisations. But the library package had a few bugs, so I modified a few files in the source code of the library and fixed them.
I have hosted my graphs on django server, and whenever I host it, the sigma package in the package.json gets loaded dynamically each time. The static library files on my machine which I had modified and fixed bugs don't get loaded. So,I get the same old package and not the modified one.
How do I access the modified library package dynamically when I host the server.
My advice is that of copying your fixed version of the library on server and install it from local path instead of remote npm repository like this:
npm install --save /path/to/fixed/lib/dir/in/server.
See this answer: npm local install
Pay attention that your fixed lib won't be sync with official one.
I don't know how you modify the library but i suggest to fork the official repository and syncronize your local one with remote one as for example explaind here sync forked repo github.
In this way you can sync to official repo while you mantain your fix and you will install your modified local one. Eventually consider to open issues and PR on sigmajs official repo to apply your fix directly to official library. If they will be accepted you can then install directly official version.
I have an NPM project that uses a git dependency
{
"repository": {
"type": "git",
"url": "https://bitbucket.org/my-private-group"
},
"dependencies": {
"my-dependency": "bitbucket:group/lib#version",
},
}
Now I want to build this project in CI using Docker with node installed.
Problem: node install tries to call git and fails because git is not there. But even if I install git it still requires authentication because it is a private repository.
At the moment I see the following solutions:
I would have to install git in docker and add an SSH key to be able to download the source code.
I may pack the related repository into the Docker image and use npm link. But this option still requires knowing dependencies set up in package.json which makes it complicated.
Setup an own npm repository to post artifact and do not use git dependencies. This option is unfortunately not achievable in my case.
Question: What is the best way of handling git dependencies in CI? Are there any other options a part from the listed options. What is the best practice?
Pulling from git without git installed is kinda hard. And installing git is easy. Just list is as a dependency for your project. This project requires windows/linux/mac os, node js, git.
You're allowing people to pull people from a private repo... that moment they have access to your source code... so all use of having the repo private is lost anyway. Anyone who wants to duplicate your code can do easily the moment it's on their computer, even if it's obfuscated.
So, I would go back a step and ask you to start asking yourself why is the repo private? Is it code that is only distributed when an NDA is present? If so, you could consider working with ssh key files to log in.
Or, you could host your files on a privately hosted gogs server, where you whitelist IP's in the firewall/nginx router that can pull from the gogs repository on your server.
If you want anyone to be able to use your repository in the final distribution of your project, you're better off lifting the private setting of your repository. You might even get some free help fixing some bugs.
I believe bitbucket has something called deployment keys which gives read only access to repositories. I am using deployment keys to build my private projects and its private dependencies.
The private key is stored in the CI server (Jenkins) and is injected into the appropriate project during its build process.
Another way is to use deployment keys with the private key stored in the project itself which can then be used during the build process.
Update
Assuming Jenkins Pipeline the following is an example of how to access ssh keys set in Jenkins using the Credentials Binding Plugin
stage('Sample') {
agent {
docker {
image 'node:12'
}
}
steps {
withCredentials([
sshUserPrivateKey(
credentialsId: 'ssh-key-name-here',
keyFileVariable: 'GIT_DEPLOY_KEY_FILE'
)
]) {
sh "cat $GIT_DEPLOY_KEY_FILE"
}
}
}
Update Sept 16th 2019
I recently came across the Build Enhancements made in Docker 18.09.
I have not yet to explore this but I think it can be used to solve the credential problem.
I am developing a package (library) at the same time as the application that uses it (also a package) and I would like to use the local library if available as sibling of the application, and otherwise download the latest release from github.
I tried having two references in dependencies and optionalDependencies, hoping that a failure of the latter would still use the specification in the former, but that doesn't work. The package is skipped.
Is this possible at all? Or maybe there are other ways to solve the problem? Maybe some creative use of the script hooks?
Maybe what you can do is to publish it on GitHub and in your package.json you can call directly from the repository something like this:
"dependencies": {
"mongoose-cipher": "git+ssh://git#github.com:estrada9166/mongoose-
cipher.git"
}
or
"dependencies": {
"mongoose-cipher": "git+https://git#github.com:estrada9166/mongoose-
cipher.git"
}
also you can specify the release, in case your repository has one, something like:
"dependencies": {
"mongoose-cipher": "git+ssh://git#github.com:estrada9166/mongoose-
cipher.git#v0.0.7"
}
you can create a private repository with your package and by this way it is safe!
To install from GitHub: npm install <git repo url>
also you can add your package on the node_modules folder and add the dependency on your package.json but personally I prefer to publish to GitHub and install it on my project, is easier.
I fixed a couple of issues in nested dependencies of my node.js project (dependencies are managed by npm). These fixes are pending pull requests and thus aren't published. What's the best way to use them in my project?
I know that I can do npm link inside the fixed version of the library and then npm link library-name inside my project to force npm to use my fixed version. This approach works but installs my library-name globally on my machine which I don't like.
Is it possible to have it locally in main project's repo, force the project to use it and don't do npm link.
You can use a url as the dependency and point it to your own repo (fork).
https://docs.npmjs.com/files/package.json#urls-as-dependencies
for example:
"dependencies": {
"foo": "git+ssh://user#hostname:project.git#commit-ish"
}
If your pull requests are on GitHub its even easier...
As of version 1.1.65, you can refer to GitHub urls as just "foo":
"user/foo-project". Just as with git URLs, a commit-ish suffix can be
included.
https://docs.npmjs.com/files/package.json#github-urls
Assumed that I have written a module for Node.js which I would like to keep private. I know that I can (should) add the line:
"private": "true"
to the package.json file, and I also know that I can npm install this module using a file system path or a link to a git repository, including GitHub.
I also know that I can put such a file system path or a link to a git repo into package.json, so that the dependencies part may look somewhat like this:
"dependencies": {
"myprivatemodule": "git#github.com:..."
}
What I now want is not to link to the latest version, but to a specific one. The only possibility I know of is to link to a specific commit using its ID. But this is way less readable and worse maintainable than using a version number such as 0.3.1.
So my question is: Is it possible to specify such a version number anyway and make npm search the git repository for the latest commit that includes this version?
If not, how do you resolve this issue in your projects? Do you live with commit IDs or is there a better solution to this?
The accepted answer did not work for me.
Here's what I'm doing to pull a package from github:
npm install --save "git://github.com/username/package.git#commit"
Or adding it manually on package.json:
"dependencies": {
"package": "git://github.com/username/package.git#commit"
}
Here's the full npm documentation:
https://docs.npmjs.com/cli/v9/configuring-npm/package-json?v=true#git-urls-as-dependencies
A dependency has to be available from the registry to be installed just by specifying a version descriptor.
You can certainly create and use your own registry instead of registry.npmjs.org if your projects shouldn't be shared publicly.
But, if it's not in a registry, it'll have to be referenced by URL or Git URL. To specify a version with a Git URL, include an appropriate <commit-ish>, such as a tag, at the end as a URL fragment.
Example, for a tag named 0.3.1:
"dependencies": {
"myprivatemodule": "git#github.com:...#0.3.1"
}
Note: The above snippet shows the base URL the same as it was posted in the question.
The snipped portion (...) should be filled in:
"myprivatemodule": "git#github.com:{owner}/{project}.git#0.3.1"
And, a different address format will be needed when SSH access isn't available:
"myprivatemodule": "git://github.com/{owner}/{project}.git#0.3.1"
Depending on your OS, you may also be able to link to the dependency in another folder where you have it cloned from Github.
If by version you mean a tag or a release, then github provides download links for those. For example, if I want to install fetch version 0.3.2 (it is not available on npm), then I add to my package.json under dependencies:
"fetch": "https://github.com/github/fetch/archive/v0.3.2.tar.gz",
The only disadvantage when compared with the commit hash approach is that a hash is guaranteed not to represent changed code, whereas a tag could be replaced. Thankfully this rarely happens.
Update:
These days the approach I use is the compact notation for a GitHub served dependency:
"dependencies": {
"package": "github:username/package#commit"
}
Where commit can be anything commitish, like a tag. In the case of GitHub you can even drop the initial github: since it's the default.
This command installs npm package username/package from specific git commit:
npm install https://github.com/username/package#3d0a21cc
Here 3d0a21cc is first 8 characters of commit hash.
My example comment to #qubyte above got chopped, so here's something that's easier to read...
The method #surjikal described above works for branch commits, but it didn't work for a tree commit I was trying include.
The archive mode also works for commits. For example, fetch # a2fbf83
npm:
npm install https://github.com/github/fetch/archive/a2fbf834773b8dc20eef83bb53d081863d3fc87f.tar.gz
yarn:
yarn add https://github.com/github/fetch/archive/a2fbf834773b8dc20eef83bb53d081863d3fc87f.tar.gz
format:
https://github.com/<owner>/<repo>/archive/<commit-id>.tar.gz
Here's the tree commit that required the /archive/ mode:
yarn add https://github.com/vuejs/vuex/archive/c3626f779b8ea902789dd1c4417cb7d7ef09b557.tar.gz
for the related vuex commit
I needed to run two versions of tfjs-core and found that both needed to be built after being installed.
package.json:
"dependencies": {
"tfjs-core-0.14.3": "git://github.com/tensorflow/tfjs-core#bb0a830b3bda1461327f083ceb3f889117209db2",
"tfjs-core-1.1.0": "git://github.com/tensorflow/tfjs-core#220660ed8b9a252f9d0847a4f4e3c76ba5188669"
}
Then:
cd node_modules/tfjs-core-0.14.3 && yarn install && yarn build-npm && cd ../../
cd node_modules/tfjs-core-1.1.0 && yarn install && yarn build-npm && cd ../../
And finally, to use the libraries:
import * as tf0143 from '../node_modules/tfjs-core-0.14.3/dist/tf-core.min.js';
import * as tf110 from '../node_modules/tfjs-core-1.1.0/dist/tf-core.min.js';
This worked great but is most certainly #hoodrat
I describe here a problem that I faced when run npm install - the package does not appear in node_modules.
The issue was that the name value in package.json of installed package was different than the name of imported package (key in package.json of my project).
So if your installed project name is some-package (name value in its package.json) then
in package.json of your project write: "some-package": "owner/some-repo#tag".
If you're doing this with more than one module and want to have more control over versions, you should look into having your own private npm registry.
This way you can npm publish your modules to your private npm registry and use package.json entries the same way you would for public modules.
https://docs.npmjs.com/files/package.json#dependencies