I would like to download node module packages (listed in a package.json file, in the present working directory) source code to a node_modules subdirectory of the present working directory, without compiling, or installing those modules. Now I have seen the related question download source from npm without npm install xxx but that question dealt with downloading the source code of individual modules specified to NPM directly (i.e., without using a package.json file). The reason why I want to do this is because I am working on developing an Atom package for the Open Build Service (OBS) of openSUSE and this seems like one of the necessary steps I need to go through in order to achieve this.
The source code is not shipped with the npm distributed code. The best you could do is read the package.json and look for the { repository: url { } } key if it exists and if it's a git repo (which most of them will be) clone it.
However be aware that the source code often requires a build step before it can be used, as in an npm prepublish step defined in the source code. In modern Javascript projects a common example of this is transpiling ES6 code to ES5 code for use in NodeJS and the browser.
I have not made an Atom package but I'm fairly certain you don't need to do any of this.
Related
I want to try and make some changes to a package published in npm? (I've suggest some changes as an issue but I think they are simple enough for me to attempt them).
https://www.npmjs.com/package/bt-presence#contributing--modifying
The author supplies some information on how to modify the package, but not really enough for someone doing it for the first time.
Where should I clone the GitHub repo to? The folder where the package is installed? I tried it in my home folder and that would not build (unmodified).
The command npm run build - where is this run from? The root folder of the package where the package.json is?
Will I need to modify the package.json?
In general what is the best way to develop something like this for npm? I've worked on packages before but they were simply Javascript.
If you want to work on the bt-presence package in isolation, you can put the cloned repository anywhere. If you want to use your modified version of bt-presence in combination with an application, my recommended approach is to register bt-presence as a dependency in the application's package.json file with the version set to a relative path to your bt-presence repository; then running npm install in the application will make a symlink from node_modules/bt-presence in the application to your bt-presence repository.
npm run build should indeed be run from the root folder that contains the package.json of bt-presence.
If you just want to change the code of bt-presence, you won't need to modify its package.json. You would only modify the package.json if you need to change any of the settings in there, e.g, if you need to add additional dependencies to your version of bt-presence.
None of the above is really specific to TypeScript. (Some JavaScript packages have build processes too if they need to transform or package the JavaScript files in some way.)
This must be a commonly solved problem, but I cannot find a whole lot on Google/SO so far.
When we run npm install and fetch say 50+ packages including devDependencies as well as runtime dependencies, npm creates node_modules (if needed) and adds each of those packages inside that folder. This means we end up with thousands of extraneous files included under node_modules. Each of those packages contains their own package.json, README.md, minified files, source files, etc. Our application really only cares about jquery.js (for DEV) and jquery.min.js (for PROD), so it seems to be a waste to include all of these other files into our build and therefore our web server.
Is there a standard when it comes to handle these npm packages in a way so that we simply expose ONLY the necessary files to the user? I imagine many people have this kind of issue but I don't see any built in npm constructs that allow us to do this easily.
See below.. the yellow highlighted files are the only files we really care about in Production, but we get all these extra files as well including the source code.
The most common solution consist of bundling your application on a different machine and then expose the built artefacts on production server.
There are a lot of great JS bundlers out there. The ones I have personally used are Browserify, Webpack, and Rollup. All amazing tools.
The main idea consists of writing your application in a Node environment and then bundle it to make it readable to the browser.
For simpler projects I find Browserify a very good compromise between power and ease of configuration. But it's a matter of taste, at the end. :)
Base on what I read about npm install documentation I do not think there is a option to manipulate the installation in the way you want. The packages will install the way the package author decides to package it, sometimes minified sometimes not.
Having said that, you should look for third party solutions like modclean which does exactly what you want post package installation. Run this command in the root of your project directory
npm install modclean -g
modclean
As long as your test coverage is good, ModClean would be perfect for your need.
Edit the package.json file and remove all the unnecessary dependencies then do
npm install --save
By doing this, it will create a local node_modules folder and only download the necessary packages into it (not the global node_modules folder)
Keep in mind, by default, node checks for local node_modules folder. If it couldn't find it, it will use the global folder.
Also, you don't expose all the packages in the node_modules folder. In fact, they will not be used unless you require(); them in the node.js file
EDIT:
For jQuery to work in Node, a window with a document is required. Since no such window exists natively in Node, one can be mocked by tools such as jsdom. This can be useful for testing purposes. https://www.npmjs.com/package/jquery
require("jsdom").env("", function(err, window) {
if (err) {
console.error(err);
return;
}
var $ = require("jquery")(window);
});
So jquery module do things a bit differently behind the scene for node.js comparing to the regular front-end jquery.
It requires jsdom so you will have to download that as well from here https://github.com/tmpvar/jsdom
I work on two repositories at once. One depends on the other (listed in package.json dependencies).
So I am using npm link ..\theOne in other to work on both modules at once. As a result I can test the modification on one module on the other. Problem is when doing npm shrinkwrap on this other module: it will generate errors like:
npm ERR! extraneous C:\other\node_modules\theOne\node_modules\{xxxx}
{xxxx} is a dev dependencies that appears as extraneous for npm.
Anyone has succeded to shrinkwrap a module with symlink to another modules?
NB:
npm 3.10.3
node 6.3.0
I post a solution here. It doesn't explain how to shrinkwrap symlinks but I found a better workaround to develop multiple modules at once with dependencies between those modules.
The solution is to use an alternative to npm link to handle the modules relations during development: Instead of having a folder A linked to another folder B (aka symlink), the solution is to have the modified files from B copied in folder A. This solution is very powerful because it avoid getting node module shrinkwrapping error due to unwanted modules from B. How to do it:
On Mac
You can use wml: Wml listens to changes in some folder (using Watchman) and copies changed files into another folder. I never use it but my teammate using Mac use it every day.
On Window
I use DSynchronize (after clicking this link, scroll down to see the executable). DSynchronize is a stand-alone utility that let you periodically synchronize two or more folders. You can exclude from copying some folders (like node_modules) or include others (like lib). The configuration file DSynchronize.ini can be edited with a text editor. For example:
Source0000=-C:\DEV\workspace\js-common
Destination0000=-C:\DEV\workspace\connexme\node_modules\js-common
Filter0000=
VolumeSerialOri0000=407325536
VolumeSerialDest0000=407325536
ExcludeFilter0000=0
NoSubDirectory0000=0
NoFilterDirectory0000=1
DateBeforeToExcludeFiles0000=00:00:00
FilterFolder0000=\.git|\.vscode|\node_modules
This configuration will copy file from C:\DEV\workspace\js-common to C:\DEV\workspace\connexme\node_modules\js-common. This way, when I npm shrinkwrap my project connexme, I'll doesn't get extraneous bad folder from js-common folder.
Just a three last things about editing DSynchronize.ini for DSynchronize configuration:
Using it or closing it will save the configuration in DSynchronize.ini so be careful when editing DSynchronize.iniand the closing it. It will override DSynchronize.ini.
Make sure each variables is unique. Example: Source0000, Source0001...
Make sure to have all the variables present for each Source/Destinations (see example above).
Is it possible to force an external npm dependency to use a different node.js package that offers the same API but a different implementation?
If you're willing to do that and that module is open source you could fork that on github, change their package.json to include the module you want and use github url for your own package.json like this:
"modulename": "git+https://git#github.com/user/repo.git"
You should be able to download the source of whatever module you would prefer and put that folder within your node_modules folder. From that point you simply require it within your Node.js app like any other NPM module.
I recommend downloading the code for the API you want, creating an src/assets folder, placing it in there, changing the package name in package.json to something not used in npm, then using 'require('newPackageName')' within your code.
If you decide to use some of package.json's capabilities to point towards a specific version (like using "1.4.7" as opposed to "^1.4.7") or if you point to a github address, be careful when you run npm update. It will replace your URL with the latest version in npmjs.org with that specific name. I don't know if it still does this in newer versions of npm, but in the version that works with Node.js 0.12, this is the default behavior.
I can tell you that node shrinkwrap will work, but it will prevent any other packages from being updated as well. No, you cannot just have one shrinkwrapped dependency, it has to be all of them, or npm update won't work.
I used the steps explained in this page: http://nodejs.org/api/addons.html and successfully created a addon.node file using the node-gyp tool. it works fine and it's a wrapper of a c++ static library.
Now I want to distribute this, I created the package.json file using:
npm init
and test it using "npm install . -g" but it tries to recompile the module which will be difficult to achieve because it will require the libraries that I'm embedding into the .node file, is it possible to distribute the .node file that I already compiled in my system?
How can I include the compiled .node file into the npm package and upload it to the npm registry. I'm sure I'm just one step to made it, but I dont know where to start.
I read about the dependencies, but seems that it's suited when your module depends on other modules, and not with your own .node file.
Thanks for your help.
ok, I finally did what I wanted to achieve, here're the options in case someone else needs this:
To avoid the compilation you could create a new folder and copy the package.json in there, along with the .node file, I didn't find this, just tried and it worked.
Provide the required libraries to allow the user his own compilation.
Although the first one worked well, I will need to create a package for windows, linux, mac, etc. Which looks very odd to say: "if you are in linux use: npm install xxx-linux", so I decided to adjust my library to allow the user the module compilation.
To do this I created a "client-dev" installer that has the required libraries precompiled, as long as the include headers required, then created the node module to be compiled using the preinstalled libraries and headers. I will need to add a help in my website to explain that, in order to install the module, the user will need to install the dependencies first using apt-get, windows installer, or mac pkg.
Although this works for me, I don't know if that will be maintainable in the long run, but I didn't find a better way to do this. (the only link that finally enlightened my goal was one saying: "if you're going to use node modules with precompiled libraries you will have nightmares", anyway... I prefer that instead of doing a full implementation in node js from scratch and maintain version for java, c#, nodejs, php, etc.