I've converted to npm for my build system: no gulp etc. Also no webpack, rollup etc, it's an es6 system based on modules & no bundling. Sure is simple!
Obviously I don't want to drag around a node_modules hierarchy for my run-time, front end modules. And don't want to import foo from './node_modules/god/awful/path.js'. So I'd like to have a top level directory for the run-time, front-end dependencies.
So how do I copy my "dependencies", not "devDependencies", to a top level directory for deployment?
I've got a run script that can do it but it's pretty messy and the location of the package under node_modules is not always obvious. Maybe there's a package for doing this automatically? Or a nifty npm trick of some sort?
OK, I'm starting to need this even more than before so thought I'd nag and be clearer about what I seem to be forced to do.
First of all, I use no workflow task managers, just npm run scripts in package.json.
My dependencies (npm --save .. not --save-dev) are:
"dependencies": {
"lzma": "^2.3.0",
"pako": "^1.0.0",
"three": "*"
},
.. and my scripts cli for hoisting the dependencies into a top level libs/ dir is simply a huge cp:
"build-deps": "cp
node_modules/lzma/src/lzma.js
node_modules/lzma/src/lzma_worker.js
node_modules/pako/dist/pako.min.js
node_modules/three/build/three.js
node_modules/three/build/three.min.js
node_modules/three/examples/js/controls/OrbitControls.js
node_modules/three/examples/js/controls/FlyControls.js
node_modules/three/examples/js/controls/FirstPersonControls.js
node_modules/three/examples/js/libs/stats.min.js
node_modules/three/examples/js/libs/dat.gui.min.js
libs/",
This is pretty primitive: I have to find the dependencies in node_modules (not always obvious) and add them to the list by hand. Sure makes me want fewer dependencies! :)
I know that bower is designed for "front end" dependencies (--save in npm speak). But it seems like npm would be perfect for dependencies, and yet it appears I need to be this primitive.
What am I missing here? What do you do?
This is something I wrestled with in my earliest forays into development with Node packages. I had the exact same idea you had: "I can use npm to pull in whatever I need for my project, great!" and then trying to manage how I reference each of my dependent libraries. I didn't properly grasp at the time that npm wasn't just there to help me fetch my dependencies, it's also there to help me manage them.
Here's the long & short of it: you do NOT want to be copying anything out of your node_modules folder to some other, more human-convenient location. There's a lot of reasons why, but the biggest is that you don't need to copy anything out of node_modules -- everything your project needs is right there.
When you're developing in ECMAScript 2015+ you should only ever have to do the following (apologies for the overly-simplistic code):
/* N.B. These all reside under node_modules, yet I don't
* have to spell out their paths under node_modules: */
import $ from 'jquery';
import _ from 'lodash';
import moment from 'moment';
import NiftyLibrary from 'niftywhatever';
// ... code ...
let $name = $('#name');
let now = moment();
// ... other code ...
In other words, your development environment setup should just handle the module resolution for you. You should never have to specify that the jQuery library you want to use is at "node_modules/jquery/dist/jquery.min.js". If you're doing this you should take a minute to figure out why you're doing this -- it's unnecessary, it's a time- and brain-suck and you'd rather be writing your application code than managing your dependencies ... NOT managing the node_modules tree.
You mentioned you're developing with ES6 modules, but not using webpack, gulp, Grunt, rollup or any other build or bundling tool. Is your project intended to run entirely in Node? I ask because the last I heard most browsers aren't quite ready to run ES6 modules natively. So how are your modules getting transpiled to ES5? Maybe you're approaching this in some novel way I haven't heard of yet, but in my experience a build or bundling tool is necessary. (Also, it's a heck of a lot of fun.)
I've used Grunt with RequireJS in the past, but am now using webpack 3 with Babel and a few additional loaders (depending on the type of project I'm working on). I use npm scripts to handle my top-level tasks (running a development server, building a finished distribution package, running tests, etc.), but I let webpack handle all the business of transpiling ES6 into ES5, translating Sass styles, precompiling Vue components, etc. It's a bit of work to wrap your mind around the webpack approach, but it's well worth the effort.
Maybe webpack doesn't fit your style -- fair enough. But there are a number of other tools you could use instead. They all require a little time to get acclimated to their approaches, but they should all be able to take care of module resolution for your dependencies. Once you get a build environment set up correctly it should cease being a visible part of your development workflow; you'll just reference your dependent libraries by their names, map them to a module-local variable and use them.
Is this helpful?
EDIT: This is webpack-specific, but there should be similar options available with other bundlers or build tools.
In webpack you can use the copy-webpack-plugin to copy npm-sourced dependencies to a separate folder. This can be useful within a service worker, for example, where the execution context is a little different.
What I would suggest is scoping your dependencies (#frontend/...) and symlinking the scoped folder to your top-level dir in a post install script (similar to how bower handled the transition to yarn: see https://github.com/sheerun/bower-away)
Example:
...
"dependencies": {
"#frontend/jquery": "npm:jquery#~3.4.1",
},
"engines": {
"npm": ">=6.9.0",
"node": ">=12.10.0"
},
"scripts": {
"postinstall": "node -e \"try { require('fs').symlinkSync(require('path').resolve('node_modules/#frontend'), 'static/bower', 'junction') } catch (e) { }\""
}
Requires NPM >= 6.9 for alias support.
FYI: aliases break audits
Related
I'm playing around with Yarn 2, and I want to do something like this.
I have a monorepo of the structure:
/
packages/
shared-ts/
package.json
src/
lib*/
app-ts/
package.json
src/
lib*/
app-js/
package.json
src/
lib*/
where lib* denotes that the folder is gitignored, but is where the compiled code will live.
In this example, I have a dependency library shared-ts that is used by two apps, app-ts and app-js.
The conventional approach
The conventional approach to configuring a monorepo like this, is that in shared-ts I would have a package.json like:
"main": "lib/index.js"
"scripts" : {
"build": "tsc"
}
Where the build script will build index.js and index.d.ts into the lib folder.
When both app-ts and app-js then resolve the package, they look in the lib folder and find the index.js and in app-ts's case - the index.d.ts.
This works fine, except that the developers need to remember to run the build script if they have made changes to shared-ts in order for the changes to propagate across.
Where this could potentially become problematic is where there are many layers of dependencies.
Attempted work around 1 - point main to src/index.ts.
I can change shared-ts package.json to
"main": "src/index.ts"
"scripts" : {
"build": "tsc"
}
This generally won't work, a plain node process won't be able to parse the syntax in the .ts file (eg. the import keyword).
Potential workaround - publishConfig
So something I'm considering, but haven't tried yet is, using the publishConfig
fields in the package.json
This field contains various settings that are only taken into consideration when a package is generated from your local sources (either through yarn pack or one of the publish commands like yarn npm publish).
"main": "src/index.ts",
"publishConfig": {
"main": "lib/index.js"
}
The idea being that:
When you publish a package to npm, lib/index.js will be used as main. 👍 code is ready for consumption, no compilation required.
If being used directly in the monorepo src/index.ts will be used as main. 😕 This kind of works as if you were running app-ts with ts-node for example.
However, where this starts breaking down is:
Running app-js in a development environment (where you don't have any additional syntax parsing set up).
Practical current best solution
My current best solution is to 'just give up on this 'no compile' aspiration' - if a developer makes changes to some code, they need to re-run build for the changes to propagate across.
How about using this?:
import someValue from 'some-package/src/index';
I can do this in my monorepo like the image below
I believe using nx will be good choice here. While it won't help you run the uncompiled code, it has pretty good features. In particular, you can automatically run the affected:apps on certain changes. For example, if you have a start command, it will run the start command for all the affected apps.
I wanted the same thing but had to compromise on the "Automatic compilation on changes" option in my JetBrains IDE.
It allows me to debug with ts-node as well as run the code using the native node binary.
I have a Monorepo under Lerna and Yarn Workspaces. The repo has packages which are published to npm and consumed outside the monorepo as well as within the monorepo. While developing in the monorepo we would like the main field of package.json for all such packages to point to the src directory, while when a package is used outside the monorepo, we would like the consumer to use the transpiled code in the dist folder.
I want this to be consistent across all uses of the packages, My current solution is to have the main field point to the dist folder. Then for each of the tools within the monorepo, namely jest, tsc, webpack, parcel I've had to come up with a different tool specific solution for aliasing the src directory instead of the dist directory. But I don't like the fact that I've had to do this work for each of these tools. It just doesn't seem scalable.
Has anybody come up with a lower level solution, where a module resolves to a different folder based on the environment?
Thank you.
If your internal code base is always transpiling the sources, why not just import { thing } from "my-package/src/main.js?
Then you can just leave the main field as dist for the consumers who ideally shouldn't have to keep track of additional paths when importing your packages.
There are a lot of details left our in your answer, but assuming you're using a single webpack/other instance to compile all your packages.
Another approach, since you're already coupled all your packages via the same compilation step, why not just use relative paths between the packages? That way you'll never have to act as a consumer but with slightly different needs.
And finally the third approach, which I think sounds a bit convoluted but should do exactly what you're asking for. Create a script that uses globby or some other npm package to grab all package.json files in your repository (excluding node_modules!). require() / iterate through these package.json manifest files and set the main field to an input value (say "dist"). Then, create two bin js files (hint: bin field) called set-main-dist and set-main-src, and possibly a third called unset-main.
Next, no matter what scripts you run in your package.json files at the root (or using lerna run), make sure to let the script look either like this:
"prebuild": "set-main-src"
or like this
"build": "set-main-src && build etc"
Hope one of these options work out for you. Remember that it's rarely worth going against the stream of usual patterns in tooling and what not. Make this one worth it.
I had exactly the same dilemma but with yarn3.
The solution importing always from source dint worked for my case, as the package itself might get published to npm too.
So after digging around I luckily found the package.json property publishConfig.main https://yarnpkg.com/configuration/manifest#publishConfig
With it I can change the main field from source to dist on npm publish.
Basically only for publishing we use a modified package.json.
Implemented in my package.json it looks like this:
{
"main": "./src/index.ts",
"publishConfig": {
"main": "./dist/index.js"
}
}
and when I run yarn npm publish or yarn pack the main field will be replaced temporary for the zip.
While all my tooling (jest and ts) can still rely on the main field pointing to the source.
I'm using webpack to bundle my typescript nodejs code.
I use webpack-node-externals to avoid errors in node_modules during the compile time.
webpack-node-externals says that, allows you to define externals - modules that should not be bundled.
But why? Webpack should bundle everything that I need to start my bundle right? It can extract and remove module that I don't use. (tree-shake for example).
If I use webpack-node-externals, then I'll have to do npm i in my prod folder to get all the dependencies.
I think this is miss the point of webpack can do. right?
I think you are correct that in your case, bundling into a single file would make more sense. webpack-node-external appears to be designed for use of NodeJS libraries, not standalone apps. From their doc:
When writing a node library, for instance, you may want to split your code to several files, and use Webpack to bundle them. However - you wouldn't want to bundle your code with its entire node_modules dependencies, for two reasons:
It will bloat your library on npm.
It goes against the entire npm dependencies management. If you're using Lodash, and the consumer of your library also has the same Lodash dependency, npm makes sure that it will be added only once. But bundling Lodash in your library will actually make it included twice, since npm is no longer managing this dependency.
As a consumer of a library, I want the library code to include only its logic, and just state its dependencies so they could me merged/resolved with the rest of the dependencies in my project. Bundling your code with your dependencies makes it virtually impossible.
I disagree with the comments that suggest Webpack was not designed to bundle Node scripts, considering that Webpack has a specific setting for just that (target). Unfortunately, there are too many third-party libraries that do not play nice with Webpack (as I just discovered today), so pragmatically speaking you're better off installing modules in the distribution folder anyway.
This is because of the binary dependency in node_modules/ as explained in:
https://archive.jlongster.com/Backend-Apps-with-Webpack--Part-I
Webpack will load modules from the node_modules folder and bundle them
in. This is fine for frontend code, but backend modules typically
aren't prepared for this (i.e. using require in weird ways) or even
worse are binary dependencies.
I went through this explanation, you can see my studies here:
https://github.com/ApolloTang/wf-backend-with-webpack-explained/tree/main/steps
This must be a commonly solved problem, but I cannot find a whole lot on Google/SO so far.
When we run npm install and fetch say 50+ packages including devDependencies as well as runtime dependencies, npm creates node_modules (if needed) and adds each of those packages inside that folder. This means we end up with thousands of extraneous files included under node_modules. Each of those packages contains their own package.json, README.md, minified files, source files, etc. Our application really only cares about jquery.js (for DEV) and jquery.min.js (for PROD), so it seems to be a waste to include all of these other files into our build and therefore our web server.
Is there a standard when it comes to handle these npm packages in a way so that we simply expose ONLY the necessary files to the user? I imagine many people have this kind of issue but I don't see any built in npm constructs that allow us to do this easily.
See below.. the yellow highlighted files are the only files we really care about in Production, but we get all these extra files as well including the source code.
The most common solution consist of bundling your application on a different machine and then expose the built artefacts on production server.
There are a lot of great JS bundlers out there. The ones I have personally used are Browserify, Webpack, and Rollup. All amazing tools.
The main idea consists of writing your application in a Node environment and then bundle it to make it readable to the browser.
For simpler projects I find Browserify a very good compromise between power and ease of configuration. But it's a matter of taste, at the end. :)
Base on what I read about npm install documentation I do not think there is a option to manipulate the installation in the way you want. The packages will install the way the package author decides to package it, sometimes minified sometimes not.
Having said that, you should look for third party solutions like modclean which does exactly what you want post package installation. Run this command in the root of your project directory
npm install modclean -g
modclean
As long as your test coverage is good, ModClean would be perfect for your need.
Edit the package.json file and remove all the unnecessary dependencies then do
npm install --save
By doing this, it will create a local node_modules folder and only download the necessary packages into it (not the global node_modules folder)
Keep in mind, by default, node checks for local node_modules folder. If it couldn't find it, it will use the global folder.
Also, you don't expose all the packages in the node_modules folder. In fact, they will not be used unless you require(); them in the node.js file
EDIT:
For jQuery to work in Node, a window with a document is required. Since no such window exists natively in Node, one can be mocked by tools such as jsdom. This can be useful for testing purposes. https://www.npmjs.com/package/jquery
require("jsdom").env("", function(err, window) {
if (err) {
console.error(err);
return;
}
var $ = require("jquery")(window);
});
So jquery module do things a bit differently behind the scene for node.js comparing to the regular front-end jquery.
It requires jsdom so you will have to download that as well from here https://github.com/tmpvar/jsdom
I know that doing something like this in package.json :
....
...
"dependencies" : {
"some-node-module" : "*"
}
is a bad idea since you're basically telling node to always update this module to its latest version, even though your code might not be able to handle any other version other than the current one for this particular module.
So I should instead do something like this :
....
...
"dependencies" : {
"some-node-module" : "3.4.1"
}
Which basically tells node to always use the version of the module that my code was built around.
Question
I have an app which I've first tested locally. The app has now been built, and using the package.json dependencies, npm has installed all of the appropriate node modules locally under my app's root folder (as opposed to globally, in some obscure folder I don't have immediate access to and which is irrelevant to this app - I simply don't like global installations of node modules - I find them to.. "abstract").
Given that all of the node modules are now installed locally isn't the node modules dependencies part in my package.json now redundant ?
I mean, what if something happens and npm is not available or the specific version of a module can't be found?
Isn't it best to be independent of dynamic node module installations and just have everything installed locally the first time without having to use the package.json dependencies ?
npm install & update
"you're basically telling node to always update this module to its latest version"
Packages won't be automatically updated. The only time the "*" will be an issue is when you are installing the project for the first time via npm install or when you manually run an update via npm update.
I personally prefer to pick a specific version of a module rather than use any wildcards, but even then there are some gotchas...which is why npm shrinkwrap exists.
npm shrinkwrap
Next gotcha:
basically tells node to always use the version of the module that my
code was built around
Sorta true. Let's say you use version 1.2.3 of your favorite module and package.json reflects that, but in the module itself is a package.json dependency on another module and that uses "*"...so when you install, the new internal dependency and the wildcard can wind up breaking the module you thought was 'locked down'.
See the gotcha? Hard coding a version controls for the top level versions but does not enforce anything beneath that...and if a module author you depend upon (or a module they depend upon) uses wildcards, you can't be 100% sure things will be copacetic.
To strictly enforce a version, you'll want to use npm shrinkwrap. (The link there to the docs provides more background, which is good to understand if your project uses more than a few very simple modules.)
And now...your question.
You say:
I mean, what if something happens and npm is not available or the
specific version of a module can't be found?
Based on the first two parts of this answer, it should now be clear that it doesn't hurt to have the dependencies explicitly listed in the package.json because node isn't checking things every time the app runs. npm uses package.json when specific actions (install, update, etc) are called but even then, it is a manual trigger.
While situations vary, there are very few that I can imagine where omitting dependencies in package.json is a good idea. If you ever wind up having to rebuild the project, you'll be in trouble. If the project is so good you want to share it, you'll be in trouble. Heck, if this is something for work and you want to go on vacation and need to install it on another machine...you'll be in trouble.
So given the fact that after the initial install, dependencies have no negative impact...use --save or add the dependencies to your package.json. Your future self will thank you. :)