Say I'm developing a large Node.js framework geared towards a specific market. People would use this framework to write and share their software, just like they would with NPM. The problem is I don't want to pollute the default NPM registry and I also want to control my own. I also want a clean repository where only modules relating to my framework would be stored.
My question, because NPM is flexible enough to let us override the default repository, what are my options for setting up an empty public NPM repository? Is there a managed host I can pay and have them set everything up for me? Is it better to manage it myself? Can I use a service like Digital Ocean or Linode? or perhaps I can just leverage a service like GitHub and bypass NPM altogether?
Related
Anyone can publish their Node.js package to open Node Package Manager (npm) pool.
Is there any security checks for published packages so that I can be sure that a new package won't contain any harmful code?
Also it's interesting to know if there is any voting system for node packages so that I can pick out the most voted package from a bunch of similar node packages?
The Node Security Platform is a tool designed to help developers do just that! You can test your project dependencies for known vulnerabilities in a variety of ways - from the command line, integrated with your CI system, or integrated with github.
The vulnerabilities are discovered via two sources:
A team of seasoned node.js security professionals actively auditing modules on npm.
Submissions from community members, which are verified by the aforementioned team.
It's also free to use the command line tool, as well as integration with open source github repositories.
If you are using npm enterprise, nsp is also partnered with npm to provide sidebar integration. This allows you to see vulnerability information right from the npme web ui, which sounds like what you are looking for.
As far as module popularity goes, there is a website that does this to some degree at nodejsmodules.org. I use it from time to time, but beware - they've got a very expired HTTPS cert.
Disclosure: I am an employee of ^Lift Security, the company behind the Node Security Platform.
You can also use https://nodesecurity.io/ to add security checks into your GitHub pull request flow.
If you perform a search on npmjs.com before pulling a module into your system the index of all packages on npmjs.com takes into account maintenance, quality, etc."
npm search is also a good option for npm package voting.
The npms analyzer continuously analyzes the npm ecosystem, gathering as much information as possible from a variety of sources, including GitHub, David and nsp. Using the collected information, a final score for each package is calculated based on four different aspects: Quality, Maintenance, Popularity, and Personalities
How can we organize our file-system and processes if there are multiple publishers on an npm module? Do we need a common repository (ex. GIT) or is there a smart way to use npm's own publishing & updating process?
The main issue I can't get my head around is that the initial publisher of the package is not able to get the latest version from within the package itself, is he? Unless he installs it as a dependency on another package and then updates & publishes from within that dependency.
It's certainly possible to use npm in this way, although it's probably not what it's intended for. The initial publisher of the package will create the necessary structure, e.g. using npm init, then do an npm publish to make it available to your other publishers.
A couple of things to consider. First, unless you have a tightly controlled system for taking it in turns to edit, you'll almost certainly get merge conflicts, where multiple publishers make changes simultaneously. A version control system like Git can help resolve these conflicts.
Secondly, you may not want your intermediate versions to be publicly available, for many reasons - it's likely you'll want to push out some (possibly incomplete) changes for your co-publishers to build on. Or you just might not want your code to be out in the wild during development. So you may wish to consider a private repository if you do go down this route - e.g. Sinopia, or one of the hosted solutions.
Hope that helps. For info, I use a combination of mercurial (for version control) and a private npm repo (sinopia).
I'm using nTwitter to access twitter api using node.
However, there is a bug in the search utility, and someone already forked and fixed it, but it wasn't pulled yet.
How can I use the fixed version s.t all my team will have the fixed version (meaning, just fixing it locally won't do the trick), but still use it as an npm module? can it be done at all?
Install that commit like this:
npm install git://github.com/AvianFlu/ntwitter#e496bc07b9d0138f65902a43bc267796ab1a74d1
Or install using package.json:
{
....
"dependencies": {
....
"ntwitter" : "git://github.com/AvianFlu/ntwitter#e496bc07b9d0138f65902a43bc267796ab1a74d1"
}
}
You can set up a private npm repository and upload your package there under a unique version, something like 1.2.5-yourcompanyname, so it'll be installed instead for everyone who is using this registry.
Pros: it works just like npm registry for everyone who is using it
Cons: it is usable only in closed groups, i.e. within a team or a company
You can set up a git dependency as damphat pointed out.
Pros: it will work for most of the people out of the box
Cons: it will require git to be installed on every user's machine
You can check it to your git repository either as a submodule or just like regular files. Use bundleDependencies when you're publishing it.
Pros: faster to install, and usually work for everyone
Cons: takes up space in git repository, longer checkouts, etc.
We went for the first solution, and usually are trying to avoid second one because we don't have git on production. Third one is quite popular as well.
I'm writing a number of pieces of code (for internal use) using node.js and want to store the modules (packaged up for npm) in a package repository for each distribution to the various machines they will be installed on.
Ideally, I'd like a solution similar to Debian's apt repositories in which I can run a private repository server and configure npm to use a list of repositories to install from (When installing "foo", if "foo" is known by my private server install it from there, otherwise install it from the public server).
However, it looks like the npm registry configuration key only accepts a single URL.
Is there a way to achieve what I want?
The closest I've been able to find have been:
Mirroring the public repository locally and adding my packages on top of it… but I don't want to keep that amount of data (2.5G and still downloading) replicated on AWS.
Hosting all my packages in git repositories and installing from there (which is more of a hassle).
Hosting static packages on HTTP (as far as I can tell, this would prevent me from automatically getting "the latest version". I suppose I could do something with symlinks, but that is still less flexible than git, requires full URLs (which need to be kept up to date), and doesn't give a searchable repository.
I just set this up for my work. Here's what I did:
Setup empty NPM registry: I followed the instructions from this fork of npmjs.org, which adds much improved documentation.
Setup Kappa: I used Kappa, a great npm proxy from Paypal. (I'm guessing they have a very similar use case to most people who want a private repository; this was exactly what I wanted).
Setup npm_lazy (optional): I wanted a nice cache of frequently used packages in case npmjs.org went down, so I added npm_lazy in front of the whole thing, as a caching layer.
Whole thing took two days(ish) to get up and running. As a side note, if you're worried about people pushing to the public registry by accident, I recommend adding this to your package.json:
"publishConfig": { "registry": "http://my-registry.example.com" },
This really is just a bit of paranoia; once you setup your npm to point to your Kappa/npm_lazy instance, Kappa handles publishing to your private repository for you.
Note: Kappa will only every publish to the first repository in it's config. If you need to publish to both your private registry, and the public, you will need to work out your own solution.
In your package.json, you can use any url that points to a valid npm packed module. I use an s3 with a bucket name that is hard to guess.
npm pack
s3cmd put *.tgz s3://path-to-your bucket
S3 is just an example, you could use any mean that can place a file on a web server, it can even be protected via basic auth.
I believe Paypal's Kappa project would suit your need.
Here is an article describing Paypal's Kraken project and how Kappa fits in.
I understand it wasn't available at the time of Quentin's question, but perhaps this will be useful for others that come along here.
npm-registry-client GitHub issue #42 lists several ways how to create your own repository mirror. namely:
https://github.com/cnpm/cnpmjs.org
https://github.com/rlidwka/sinopia
https://github.com/rlidwka/sinopia#similar-existing-things
https://github.com/davglass/registry-static
Overall it seems to me that you can get best answers by searching through issues in repositories owned by https://github.com/npm or by asking your question there
Based only on listening to a recent episode of NodeUp (#37?), I think you may want to have a look at irisnpm. From what I remember it's a service which gives you a merged set of the public modules and your own private modules.
As Dominic Barnes suggested, we can replicate only _design documents (CouchDB table schemes)
How to replicate design documents only?
However, it needs to check if some data needed.
You could replicate the modules you need and then write a proxy server which looks for a module in your replication. If a module doesn't exist, it could pipe the request to NPM and return the result from there.
Is it normal or hopefully best practice to use package.json in my project even though my project isn't a module/package? I'm currently using package.json for info, version, dep management and I include use 'private: true'
Many people are using package.json in applications these days. npm is still a great way to manage your dependencies while developing. As mentioned there are a few ways to ensure that your app doesn't get pushed to the public repository. And you still get the benefits of all the npm utilities.
If by "best practice" you're asking if there is a good reason not to do this, the answer is no. You should go for it.