When pointing to a different location of a package my app needs, I can usually just replace the version with "depName": "githubUser/repoName" and it works great.
But what should I do if the package I need to customize is itself a node module of some other package? e.g.
myapp
|_
express.js
|_
send.js
I prefer to avoid publishing a fork of the first package, express in this example, just to use a modified version of send.
Related
I'm new in node world and I'm in facing with a problem,
Currently I'd made a split in the project (in ts) where I work in two new packages(using ts also), to be shared between applications.
All this code are in the same repo, popularly called "monorepo". Below I'll illustrate how this are joined.
But even using the npm link when we made the transpile, from this ts to js these code wasn't included in the final build, to solve this I created two npm packages.
Therefore, apparently this have been adding more complexity into development, because when we need change something in these packages, we need do a link between these package and the project and then if we made some new modification in the packages, it's needed run the npm build in these packages to update the source to be used, because I import the package transpiled (I have some issues regarding the es6 and your imports, node don't accepted it).
I would like how can I improve this method to be less bureaucratic and more effective.
To illustrate my current situation:
/Repo
|_ _/my-app
| |_ _/node_modules
| | |_ _/package-A (npm package)
| | |_ _/package-B (npm package)
|_ _/package-A
|_ _/package-B
|_ _/others-apps
I have a monorepo that has many micro-services in it. There are some library-type functions / classes that I want to make available to any micro-service that needs it. However, if that library package declares a peer dependency, the peer dependency is not found when running code from within the thing that depends on the library.
Consider this repo structure:
lib
some-library (peerDepends on foo)
index.js (requires foo)
node_modules will be empty
services
some-service (depends on foo, and some-library)
index.js (requires some-library)
node_modules will have:
foo
some-library will be a symlink to ../../lib/some-library
When running node services/some-service/index.js, you'll get the error "Cannot find module 'foo'", emanating from lib/some-library/index.js.
Presumably this happens because node is only looking at lib/some-library/node_modules and any node_modules folder that is in an ancestor directory. But since this code was run from services/some-service (as the working directory), and because of the symlink in services/some-service/node_modules, I would've expected this to work.
Here's a repo you can easily clone to see the problem: https://github.com/jthomerson/example-local-dependency-problem
git clone git#github.com:jthomerson/example-local-dependency-problem.git
cd example-local-dependency-problem
cd services/some-service
npm install
node index.js
I only see two solutions:
Don't use peerDependencies inside the library
Install each peer dependency at the root of the project for the sake of local development and testing.
Neither of those is a real great solution because it doesn't allow each service to have different versions of the dependencies, and thus means that if the local version (or the library's version) of a dependency is bumped, all services that uses the library then have their dependencies version bumped at the same time, which makes them more brittle because they're all tied together.
How about adding the --preserve-symlinks flag?
E.g.:
node --preserve-symlinks index.js
Here's a link to the docs
I had a dual-workspace setup in which:
workspace1
shared-library
module-library (peer depends on shared-library)
workspace2
main-app (depends on module-library and shared-library)
Now dependencies for workspace projects are defined in the tsconfig.base.json file under compilerOptions.paths.
However for workspace2 which is unrelated to workspace1, I install the packages (both via file:. When I then build main-app I get the error that module-library is unable to find shared-library (even though it's installed in workspace2.
I had to add ./../workspace1/dist/shared-library to the compilerOptions.paths in tsconfig.base.json of workspace2 (note the reference to workspace1).
This obviously couples the workspaces on my filesystem. But for development purposes this is perfect.
This must be a commonly solved problem, but I cannot find a whole lot on Google/SO so far.
When we run npm install and fetch say 50+ packages including devDependencies as well as runtime dependencies, npm creates node_modules (if needed) and adds each of those packages inside that folder. This means we end up with thousands of extraneous files included under node_modules. Each of those packages contains their own package.json, README.md, minified files, source files, etc. Our application really only cares about jquery.js (for DEV) and jquery.min.js (for PROD), so it seems to be a waste to include all of these other files into our build and therefore our web server.
Is there a standard when it comes to handle these npm packages in a way so that we simply expose ONLY the necessary files to the user? I imagine many people have this kind of issue but I don't see any built in npm constructs that allow us to do this easily.
See below.. the yellow highlighted files are the only files we really care about in Production, but we get all these extra files as well including the source code.
The most common solution consist of bundling your application on a different machine and then expose the built artefacts on production server.
There are a lot of great JS bundlers out there. The ones I have personally used are Browserify, Webpack, and Rollup. All amazing tools.
The main idea consists of writing your application in a Node environment and then bundle it to make it readable to the browser.
For simpler projects I find Browserify a very good compromise between power and ease of configuration. But it's a matter of taste, at the end. :)
Base on what I read about npm install documentation I do not think there is a option to manipulate the installation in the way you want. The packages will install the way the package author decides to package it, sometimes minified sometimes not.
Having said that, you should look for third party solutions like modclean which does exactly what you want post package installation. Run this command in the root of your project directory
npm install modclean -g
modclean
As long as your test coverage is good, ModClean would be perfect for your need.
Edit the package.json file and remove all the unnecessary dependencies then do
npm install --save
By doing this, it will create a local node_modules folder and only download the necessary packages into it (not the global node_modules folder)
Keep in mind, by default, node checks for local node_modules folder. If it couldn't find it, it will use the global folder.
Also, you don't expose all the packages in the node_modules folder. In fact, they will not be used unless you require(); them in the node.js file
EDIT:
For jQuery to work in Node, a window with a document is required. Since no such window exists natively in Node, one can be mocked by tools such as jsdom. This can be useful for testing purposes. https://www.npmjs.com/package/jquery
require("jsdom").env("", function(err, window) {
if (err) {
console.error(err);
return;
}
var $ = require("jquery")(window);
});
So jquery module do things a bit differently behind the scene for node.js comparing to the regular front-end jquery.
It requires jsdom so you will have to download that as well from here https://github.com/tmpvar/jsdom
I am trying to figure out an effective way to bundle and distribute various dependencies (node modules and/or "client"-side scripts and framework like Angular) with my Electron App.
Although the basic approach of npm install module-name --save works well for development, it is not so good in the end when it comes to minimizing the size of your app and using minified resources at runtime. For instance, virtually all npm packages (including node modules) come with a lot of "extra baggage" like readmes, various versions of components (minified, not minified, ES2015, no-ES2015, etc). While these are great for development, all these files have absolutely no need to be included in the version you will be distributing.
Currently there seem to be 2 ways to sort of address the problem:
Electron Builder recommends using 2-file package.json system.
Any dependency that is used during development only should be npm-installed using --save-dev and then prunning should be used when building the app for distribution.
In that regard I have several questions:
I am not quite sure why there is a need for 2-file package.json system if one can install dev-only modules/ dependencies with --save-dev and then use pruning during the actual app build/compilation?
Regardless of which method above is used, you still end up with full npm packages in your app, inclduying all the miscellaneous/duplicated files that are not used by your app. So how does one "prune" so to speak the npm packages themselves so that only the actual files that are being used at run-time (like minified scripts) get included?
Will using Bower for "client-side" packages (like AngularJS 2, Bootstrap, jQuery, etc.) and using npm for node modules (like fs-extra) be a better option in as far as separation of concerns and ease of bundling later?
Could WebPack be used to produce only the needed files, at least for the "cient-side", so that only real node modules will be included with the app, while the rest of it will be in the form of web-pack compiled set of files?
Any practical tips on how this bundling of dependencies and distribution should be accredited out in practice? Gulp-scripts? Web-pack scripts? Project structure?
Thank you.
I am still in the learning curve of adopting the best practices in code deployment. But here is my starting list of what is recommended.
Yes, npm install --save-dev is the first easiest thing to isolate dev and build specific packages. This includes gulp/grunt/webpack and its loaders or additional packages. These are used only for building and never in the code that actually is run. All packages used by the app should be installed with npm install --save so that it is project level available. So, in production, you would no npm install --production in machines which will not install dev packages at all. See What's the difference between dependencies, devDependencies and peerDependencies in npm package.json file? for more info.
While the original recommendation was to use bower for client side and npm for server side, both can be installed using npm too. After all, both does the same job of managing the packages and dependencies. However, if web pack is used, it is recommended that npm is used for client side dependencies also.
package.json should be thought of managing the dependent packages only and not for building. For building and picking only the required files, you need task runners like gulp/grunt or bundlers like web pack.
While gulp/grunt is very popular for build automation which includes bundling all dependent javascript in file and minifying them in to one file, webpack/browserify is a better option as it supports module import. Module import is intuitive way of require one module in another in node js type of coding
var util = require('./myapp/lib/utils.js') This is powerful way of mentioning the required dependencies in the code. The web pack builder runs like gulp as build process. But instead of looking through html file for all js files, looks at starting js file and and determines all dependent code mentioned by the require statements recursively and packages accordingly. It also minifies the code. It also loads css and image files in one bundle to reduce server trips. If needed, some modules can be configured to be loaded at runtime dynamically further reducing page load. NPM vs. Bower vs. Browserify vs. Gulp vs. Grunt vs. Webpack discusses this at length.
Webpack can be used to bundle client side app optimally while server side need not be bundled or minified as there is no download.
In web pack, though you can mention dependent modules with lib file path, the recommendation is to npm install all dependencies and mention the module name. For example if you have installed jquery, instead of giving path like /libs/jquery.min.js, you can mention as 'jquery'. Webpack will automatically pull the jquery lib and dependencies and minimise it. If they are common modules, it will be chunked too. So, it is better to npm install dependent packages instead of bower install.
ES2015 provides lot of benefits during coding time including type checking and modules. However all browsers do not yet support the spec natively. So you need to transpile the code to older version that browsers understand. This is done by transpilers like Babel that can be run with gulp. Webpack has in-built babel loader so web pack understands ES2015. It is recommended to use ES2015 module system as it will soon become the defacto way of coding and since there is transpiler, there is no worry of this not being supported in IE8/9.
For project structure you could have
server
client
src containing js files
dist containing html and build files generated
webpack.dev.config.js and webpack.prod.config.js can be at root level.
I have found that this area is an ocean and different schools of best practices.This is probably one set of best practices. Feel free to choose the set that works for your scenario. Look forward for more comments to add to this set.
Is it possible to force an external npm dependency to use a different node.js package that offers the same API but a different implementation?
If you're willing to do that and that module is open source you could fork that on github, change their package.json to include the module you want and use github url for your own package.json like this:
"modulename": "git+https://git#github.com/user/repo.git"
You should be able to download the source of whatever module you would prefer and put that folder within your node_modules folder. From that point you simply require it within your Node.js app like any other NPM module.
I recommend downloading the code for the API you want, creating an src/assets folder, placing it in there, changing the package name in package.json to something not used in npm, then using 'require('newPackageName')' within your code.
If you decide to use some of package.json's capabilities to point towards a specific version (like using "1.4.7" as opposed to "^1.4.7") or if you point to a github address, be careful when you run npm update. It will replace your URL with the latest version in npmjs.org with that specific name. I don't know if it still does this in newer versions of npm, but in the version that works with Node.js 0.12, this is the default behavior.
I can tell you that node shrinkwrap will work, but it will prevent any other packages from being updated as well. No, you cannot just have one shrinkwrapped dependency, it has to be all of them, or npm update won't work.