Recently I started to use a private npm packages for backend.
They stored on our private packages server (an npm repository I deployed among application components.
And while I mostly like how it all works and happens, but there is a gotcha.
Imagine a packages like:
#example/logging, #example/utilities, #example/validation, #example/database, #example/security - low-level packages. And many of these low-level packages depends on #selfcad/logging package.
#example/updates-queue, #example/subscriptions, #example/auth-tools, #example/cloud-functions and #example/backend-core packages are representing high-level abstractions and components. Each of high-level packages depending on few or sometimes all of low-level packages.
Some of these packages are referenced in cloud functions via #example/cloud-functions package, especially low-level packages. Other used directly in a cloud functions.
Plus there is a backend component where most of packages are used directly.
So, true headache starts, when a change must be made e. g. in the low-level package like #example/logging or #example/validation.
And what is the most easier way to spread the update or fix to widely referenced lower package? With nested dependencies.
Like so: a package, e. g. #example/validation referenced directly and indirectly, like #example/configuration uses #example/validation, #example/database uses #example/validation and I updated #example/validation internals. How to spread such update across packages?
Maybe you should have a look to Yarn Workspaces. This is a useful feature of yarn to link dependent packages each others using a mono repo. This feature also includes auto-discovery/auto-linking.
Related
My Node application needs to be deployed on Windows and Linux. The main deployment package is built on a Linux CI server.
When this package is deployed to Windows, it crashes immediately due to missing native bindings, such as those for sqlite. Only the bindings for the build platform (Linux) are restored.
With a deadline approaching, we just set up a Windows build configuration which outputs a Windows specific package that contains the appropriate bindings, and we choose the appropriate artifact to bundle in the installer.
This works but feels fragile, as we would need to keep the Node versions in sync between the two otherwise unrelated environments. I would like to be able to do this with a single build configuration.
I couldn't find any guidance on how this is done. I'm imagining a command-line option like --platform=windows to npm ci, or a modification to package.json but I couldn't find any information about this. Presumably this is a reasonably rare requirement, and perhaps there is no tooling around this, which would be a shame.
Another requirement is that the application must be installed without an internet connection. We cannot run npm ci or npm install when we install it as some of our clients do not permit their servers to access the public internet.
Based on your requirements it sounds like building a package on each required platform would be the safest bet, with the least number moving parts to go wrong.
As the comments have suggested most projects rely on an npm install on the required platform so you are stepping into not that common territory.
This works but feels fragile, as we would need to keep the Node versions in sync between the two otherwise unrelated environments. I would like to be able to do this with a single build configuration.
Node uses NODE_MODULE_VERSION (displayed on the releases page) to track ABI compatibility for native modules. This only changes with a new major Node release number.
The CI builds would need to create app packages for each major version of Node you run on each platform. Keeping the Node.js major versions in sync for the application a good thing in any case. Running Node N and N-1 builds until that can be achieved is good cover and probably the best option with the air gap requirements.
NPM Cache
If the air gapped clients are largely on common networks, an NPM cache/proxy (nexus/verdaccio) may be of use. The NPM cache will need a process to snapshot the repo after a production npm install on all required platforms, to be pushed out to your endpoints. Unfortunately binary modules are often distributed out of band from NPM so won't be stored in regular NPM caches. Each client instance will need a complete build environment to build any native modules from source which can sometime present it's own difficulties on Windows platforms.
Alternatives
Node.js is not a great platform for distributing packaged applications to many diverse clients, especially if you need to distribute Node itself. Any language with an external VM requirement presents difficulties. Nodes package management choices and reliance on native modules exacerbate this.
I've given up in the past and converted clients (albeit thin) to Go, as it lends itself to cross platform distribution a lot better by removing the external runtime requirement and having less variables.
I understand that packages hold several modules, but I'm starting to get confused as to if packages and modules are dependencies. Also, libraries to me maybe seem like packages you install via NPM, Nuget, RubyGems, Bower, Homebrew, or Chocolatey. So are libraries packages? Dependencies are something you need to load within your application, to have a certain functionality, but aren't some libraries(jQuery) seen as a dependency? So yea, what are the differences between these concepts?
Libraries are just a bunch of code anyone can use. For example, React.js is a JavaScript library for building front end components.
If I decide to use this library in my app, then React will become one of the modules (aka an installed instance of the library) that my app depends on. So dependencies are pretty much all of the libraries your app depends on, in order to run the way you expect it to run.
I asked the same question you did about dependencies, and I learned that it's a matter of understanding how these terms relate to one another rather than finding isolated definitions for each of them.
Hope this helps!
Basically a package is a pack with some features which fullfills some functionality of your app.
Once you install any package using npm then the package is installed as a dependency in your app inside your package.json file along with its modules(aka libraries consist of classes) stored inside node_modules folder.
I hope its clear now.
Is there a way to install meteor packages globally?
So, having the once globally installed packages installable without internet connection for projects created later, avoid repetitive downloading, and other benefits one may imagine.
Like in Node.js, using npm command (of Node Package Manager) with -g flag, npm install -g, doing so npm installs node packages into a global directory and when wanted to be loaded from javascript programs, loading from there if available, as well as looking in and loading packages from project's node modules folder.
Meteor already downloads packages into a global repository that all your local apps benefit off of.
So if you meteor add iron:router#1.0.7 it is downloaded and added to your project. Next time another project requires the same version, it is used off that same spot.
Also, there is a PACKAGES_DIR environment variable, when set, allows you to keep your own local packages centrally, so that you can share them among projects. In fact, you can keep that on a network drive (NFS) which your whole team can mount and consume centrally.
Yet, there is an inherent problem. Meteor's version resolver looks up for updates unless you pin down your package dependency versions so that is exactly why meteor seems to be so desperate to be connected.
Even if you pin your dependencies, the packages you depend on may not have (which apparently is the case for most packages) so Meteor keeps looking for updates to the whole package tree and downloads those that it deems satisfying the version constraint resolver.
The good news is, they are constantly improving their tooling, requiring lower number of lookups, faster builds, better search etc.
All in all, in essence, there is not much you can do unless Meteor provides some way of hosting an entire mirror of its package repository for you to consume offline. And I guess it is very unlikely to happen.
Meteor is a tool for the connected world and it does assume your connectivity. Heck, the whole journey begins with a curl https://install.meteor.com/ | sh
And yes, it would be great if we could hack away on a remote beach, or the 12 hour flight to that beach.
Until then, happy coding online ;)
Practically speaking, are these essentially synonymous? Or is there something I'm missing? I've use Composer (PHP), CocoaPods (Objective-C), and Bundler (Rails). I believe they describe themselves as dependency managers but can they also be consider as package mangers?
I'd say yes. Given that the javascript community calls their version of those tools (NPM and bower) "package managers", I think that the development community has essentially synonymized those terms.
EDIT I'm going to backtrack a bit. In general, I think the term package manager has to do with the delivery and installation of third party code. That said, npm is correctly named the node package manager. As I see it, a dependency manager is a different thing. It's an runtime orchestration tool. For example, there are dependency managers that simply run in the browser to load asset files in the proper order (think requireJS, browserify, cartero, etc... - or think a Dependency Injection container in say Symfony2 or Laravel) but you wouldn't call those package managers. A package manager would be something like Debian's dpkg or the node community's bower, which actually downloads third party libraries for you (that aren't currently in your software suite). Now, I think the burred lines come in when package managers decided to be smart enough to resolve version numbers for us. Because tools like npm make sure that each piece of software we declare has all of the proper versions of it's dependencies (by downloading a chain of dependencies for us), we want to call it a dependency manager. But I think it's more proper to say that it's a package manager that happens to do version resolution. It's really more of a delivery mechanism than a runtime tool, though.
All that to say, I'd like to hear what others have to say about this.
No, they are not synonyms. Look At that answer for their difference
https://stackoverflow.com/a/27290095/4016254
I would like to know the difference between the 2 methods in terms of how dependencies are handled, ease of use and configurability.
If you install from source, you lose ease of use (you have to follow a procedure yourself instead of clicking a button to install) and you have to resolve your own dependencies in many cases, whereas a package manager would do this for you in most cases. However, you gain a lot of configurability (pick and choose versions, where to install, even modify source).
Also, there's a lot of stuff out there that you can only get if you install from source, because packages haven't been made (or kept up to date), especially if you use one of the less popular package managers.
What about building software from source with the help of a package manager? Think gentoo's emerge. You see, the two choices you present are not necessarily mutually exclusive alternatives.