I am developing a very complex app that is using internally developed, open source NPM modules.
I often need to change one of those modules (extra features, bug fixing, etc.) in order for the main application to work.
At the moment, I have:
A directory called my_modules, each containing a git repository one for each module. For example module1, module2.
A directory called my_apps, where for example there is app1 which has module1 as a dependency
Under my_apps/app1/node_modules I have module1 and module2, installed via NPM
In the server, deploy by pulling the git repository, running an npm install and npm dedupe, and running the server with forever.
At this stage, if I have to fix something in one of the modules, I:
Fix it within my_apps/app1/node_modules/module1 (not git)
When it's all working, COPY the files over to my_modules/module1 and do a git push and npm publish
The server will pull the latest modules after deploy thanks to npm install
This is way, way less than ideal. It's just too error-prone. However:
Having a symbolic link link my_apps/app1/node_modules/module1 => my_modules/module1 means that module1 will look for dependencies in its own path, which often causes problems (for example, I need to make sure that EVERY module uses the same copy of module1, which is imperative)
Having a git repo under my_apps/app1/node_modules/module1 feels dangerous, in case I accidentally overwrite changes using NPM on the module. Also, once fixed the change in the local git repo, I would still then need to pull the changes in my_modules/module1. Yes a step forward from copying files over...
What's the "recommended" way of dealing with this? Any best practices?
Related
I want to try and make some changes to a package published in npm? (I've suggest some changes as an issue but I think they are simple enough for me to attempt them).
https://www.npmjs.com/package/bt-presence#contributing--modifying
The author supplies some information on how to modify the package, but not really enough for someone doing it for the first time.
Where should I clone the GitHub repo to? The folder where the package is installed? I tried it in my home folder and that would not build (unmodified).
The command npm run build - where is this run from? The root folder of the package where the package.json is?
Will I need to modify the package.json?
In general what is the best way to develop something like this for npm? I've worked on packages before but they were simply Javascript.
If you want to work on the bt-presence package in isolation, you can put the cloned repository anywhere. If you want to use your modified version of bt-presence in combination with an application, my recommended approach is to register bt-presence as a dependency in the application's package.json file with the version set to a relative path to your bt-presence repository; then running npm install in the application will make a symlink from node_modules/bt-presence in the application to your bt-presence repository.
npm run build should indeed be run from the root folder that contains the package.json of bt-presence.
If you just want to change the code of bt-presence, you won't need to modify its package.json. You would only modify the package.json if you need to change any of the settings in there, e.g, if you need to add additional dependencies to your version of bt-presence.
None of the above is really specific to TypeScript. (Some JavaScript packages have build processes too if they need to transform or package the JavaScript files in some way.)
I've pulled down a node module using NPM, and added it to package.json. However there was a need to change some of the module's code as it didn't meet my requirements 100%.
Typically when I'm working with node and git I would ignore the node_modules directory and use npm install when deploying to a server.
I'm wondering what best practise would be in my scenario, is there a way of defining a module in package.json that should be ignored if it already exists locally when running npm install? Is this already the default behaviour for all modules? How would that work if someone ran npm update? I would assume the latest version of that module would be pulled down and would overwrite my changes?
Alternatively I've thought about forking the original git repo for the module, republishing my fork to NPM and then using that instead.
Tips and ideas would be greatly appreciated :)
Alternatively I've thought about forking the original git repo for the module, republishing my fork to NPM and then using that instead.
You have the right idea here. Under NPM, you definitely don't want to split your concerns between hosted and version control-tracked resources. Fork the repo, and then answer this question: if you add the functionality to the existing module, is the pull request likely to be merged and published to NPM soon enough for you?
If the answer is no because the functionality doesn't meet the intentions of the original module, you're better off creating your own, making sure to note your fork in the README.
If you're waiting on the PR, you have an option in the interim. NPM lets you link directly to your fork's .git file.
I have a custom npm module that I am working on, and it has a GitHub repo. I'm also working on a project that uses the custom module. When working on the larger project, it is nice to use npm link so I can make changes to the module and see them right away in the main project.
To deploy to staging or production, I use shrinkwrap and shrinkpack so I can do an npm install after every deploy (some of the dependencies need binaries, and dev systems aren't the same as production systems, so they do need to be installed and not just kept in source control). Edit: I'm crossing this out as the answer below technically solves my issue, even though it doesn't solve for this particular point, but that wasn't as important as the rest of it.
Of course, since the module is linked to my main project and not listed in package.json, a deploy and install misses it entirely. I can go ahead and list it in package.json and have it point to the appropriate GitHub repo, but then every time I need to test a change in the main project I would have to commit and push those changes, then update the main project, kill and restart the app...that would get tiresome pretty quickly.
I guess I need something like the opposite of "devDependencies"; something where I can have it not install the module on dev, but do install it from GitHub when doing npm install on staging or production. Other than remembering to manually change package.json every time I need to go back and forth, is there a better way to do this?
you can specify a github repository as your package to install, in your package.json file:
{
dependencies: {
"my-library": "githubusername/my-library"
}
}
this will work in your production environment.
in your development environment, use "npm link".
from within the "my-library" folder, run npm link directly. that will tell npm on your local box that "my-library" is avaialable as a link.
now, in your project that uses "my-library", run npm link my-library. this will create a symlink to your local development version of "my-library", allowing you to change code in that repository and have it work in your other project that needs it.
once you are ready to push to production, push "my-library" to your github repository, and then you can npm install on your servers, like normal.
I'm developing a node project which depends on several npm modules. What I'm currently doing is committing all those modules to my git repository, pulling on my server, and then rebuilding the modules on the other end because the system architectures are different. What I would like to do instead, is just commit enough stuff so that npm knows what it needs to rebuild, and nothing more.
I would hoping I could do this without actually committing all the module files. I know little about npm's internals, but see that every module has a package.json file, is this all that npm will need? And if that's the case, how would I go about ignoring all files in my node_modules folder except the package.json files?
Thanks.
Why can't you just have a package.json file in your main application root listing all your dependencies? This file should then be the only thing checked in to source control.
At build time or when other developers pull the code, npm install should be run from the same directory that has your package.json file. It will then pull down all your dependencies locally.
What would be the preferred way to handle internal modules within a node.js application?
I mean, currently we have a fairly big application and there are several dependencies, installed via npm. We have node_modules in .gitignore. Therefore it's not in the repository. We use npm shrinkwrap to freeze the modules versions, so, upon deployment, an npm install puts together eveything we need into node_modules.
The problem is, since out app is getting bigger, we want to split it to smaller modules. Now, if I create a foo module and put it in node_modues, I need to allow it in the repo, which does not seem so neat to have both ignored and checked out modules in node_modules.
We can not publish these on npm registry because they are not really "public".
Is there any obvious solution that I'm not aware of?
I agree with your aesthetic of not mixing 3rd-party non-repo stuff with your in-house revision-controlled contents.
The NODE_PATH search path variable can help you here, you can use a different directory with contents under revision control.
http://nodejs.org/api/modules.html
Depending on your appetite for flexibility vs complexity, you can consider a build step that actually copies your various node modules from wherever you keep them, into an output directory (probably "named node_modules" but not necessarily).
A common solution is to store the modules in a private Git repository somewhere, and then use a git:// URL as a dependency; npm knows how to clone repositories as dependencies.