I'm maintaining some large NodeJS applications(micro-services, apps), which consume many(at least 100+) dependencies.
Updating those dependencies to fix a security vulnerability forces me to spend valuable development time.
Is there any way to automate package updates across repositories?
We're using GitLab for SCM.
I had thought of some options like:
Separating out core dependencies into a base level library.
Maintain a repository that specifies security-cleared dependency versions and repositories to be updated whenever the cleared list changes.
Any additional suggestions/examples?
Just to help shortlist approaches mentioned in the question:
1. Separating core dependencies into a "Core" library.
This can reduce the number of updates. But the libraries based on "Core" sill still have their own dependencies, which may be prone to vulnerabilities and the update cycle must be repeated for them manually. This approach just procrastinate the manual updates.
2. Maintain a repository that specifies security-cleared dependency versions and repositories to be updated whenever the cleared list changes.
Couple of options here:
Article on automating via pipeline scripts
GitHub: Use next-update + Hub CLI
GitLab: Use next-update + Lab CLI
Implement dependabot-core(GitHub) or dependabot-gitlab(GitLab)
Implement renovate.
Related
How can I share components across multiple react projects without having to publish them on a public package manager like NPM?
Option 1: You can use npm and use private packages so they're not external facing. There are also artifactories and scoped packages that usually represent company-wide projects that can be public or private. See https://docs.npmjs.com/private-modules/intro and https://docs.npmjs.com/misc/scope.
Option 2: Essentially, you can develop projects with a flattened structure. You can then import various projects and/or components into other projects or folders. This is entirely dependent on your codebase and configuration. With this model though, a lot of times publishing to npm comes fairly naturally since each folder may be its own project with its own package.json.
Updated:
Option 3: Bit focuses on the composability of components from everything from the little things like a button to the actual view and app itself—each target is its own package. Overall, it's an opinionated, yet customizable framework that can enable quicker development, managed dependencies, and organized code.
Option 4: RushJS is a monorepo manager built by Microsoft that allows for flexibility of different kinds of apps and services utilizing pnpm underneath (as opposed to yarn and npm), which alleviates problems that stem from dependency issues.
Check out Bit:
Bit is an open-source cli tool for collaborating on isolated components across projects and repositories. Use Bit to distribute discrete components from a design library or a project into a standalone reusable package and utilize it across applications.
You could also upload them to a private git repo such a Github and then pull them in from there.
Ryanve has a nice example over here: https://stackoverflow.com/a/28729646/1592783
You could create a repo of shared components and then have your Node.js start script call a shell script to do a git pull from that repo and the move the shared components from that directory to your project's directory. That way, every time you call run 'npm start' you will have the latest version of the shared components loaded into your project
I lately helped out on a project, where I added a really small dependency - in fact, it only contained a regular expression (https://www.npmjs.com/package/is-unc-path).
The feedback I got from the developer of the project was that he tries to minimize third-party dependencies if they can be implemented easily - whereby he - if I understand it correctly - asks me to just copy the code instead of adding another dependency.
To me, adding a new dependency looks just like putting some lines of code into an extra file in the repo. In addition, the developers will get informed by an update if the code needs a change.
Is it just a religious thought that drives a developer to do this? Are there maybe any costs (performance- or space-wise, etc) when adding a dependency?
I also had some disputes with my managers once concerning the third party libraries, the problem was even greater he got into believing that you should version the node_modules folder.
The source of any conflict usually is the ignorance.
His arguments were:
you should deliver to the client a working product not needing for him to do any other jobs like npm install
if github, npm is down in the moment when you run npm install on the server what you will do ?
if the library that you install has a bug who will be responsible
My arguments were:
versioning node_modules is not going to work due to how package dependencies work, each library will download his own node_modules dependencies and then your git repository will grow rapidly to hundreds of mb. Deploy will become more and more slow, downloading each time half a gb of code take time. NPM does use a module caching mechanism if there are no changes it will not download code uselessly.
the problem with left-pad was painfull but after that npm implemented a locking system and now for each package you just lock to a specific commit hash.
And Github, and npm does not have just a single instance service, they run in cloud.
When installing a dependency you always have some ideas behind and there are community best practices, usually they resume to: 1. Does the repo has unit tests. 2. The download number 3. When was the latest update.
Node.js ecosystem is built on modularity, it is not that node is so popular cause of some luck, but cause of how it was designed to create modules and reuse them. Sometimes working in node.js environment feels like putting lego pieces together and building your toy. This is the main cause of super fast development in node.js. People just reuse stuff.
Finally he stayed on his own ideas, and I left the project :D.
How can we organize our file-system and processes if there are multiple publishers on an npm module? Do we need a common repository (ex. GIT) or is there a smart way to use npm's own publishing & updating process?
The main issue I can't get my head around is that the initial publisher of the package is not able to get the latest version from within the package itself, is he? Unless he installs it as a dependency on another package and then updates & publishes from within that dependency.
It's certainly possible to use npm in this way, although it's probably not what it's intended for. The initial publisher of the package will create the necessary structure, e.g. using npm init, then do an npm publish to make it available to your other publishers.
A couple of things to consider. First, unless you have a tightly controlled system for taking it in turns to edit, you'll almost certainly get merge conflicts, where multiple publishers make changes simultaneously. A version control system like Git can help resolve these conflicts.
Secondly, you may not want your intermediate versions to be publicly available, for many reasons - it's likely you'll want to push out some (possibly incomplete) changes for your co-publishers to build on. Or you just might not want your code to be out in the wild during development. So you may wish to consider a private repository if you do go down this route - e.g. Sinopia, or one of the hosted solutions.
Hope that helps. For info, I use a combination of mercurial (for version control) and a private npm repo (sinopia).
In doing research on whether Node's node_modules should be checked into your version control repository, the most recent consensus seems to be that you should include it for applications you deploy.
Sources:
Check in node_modules vs. shrinkwrap
Should I check in node_modules to git when creating a node.js app on Heroku?
https://www.npmjs.org/doc/misc/npm-faq.html#Should-I-check-my-node_modules-folder-into-git
In reading these arguments, it lead me to question whether Composer's /vendors directory should also be checked into version control. Composer's documentation suggests that you don't:
Should I commit the dependencies in my vendor directory?
The general recommendation is no. The vendor directory [...] should be added to .gitignore/svn:ignore/etc.
The best practice is to then have all the developers use Composer to install the dependencies. Similarly, the build server, CI, deployment tools etc should be adapted to run Composer as part of their project bootstrapping.
While it can be tempting to commit it in some environment, it leads to a few problems:
Large VCS repository size and diffs when you update code.
Duplication of the history of all your dependencies in your own VCS.
[...]
Contrasting that argument is this one (source):
Doesn’t checking in node_modules create a lot of noise in the source tree that isn’t related to my app?
No, you’re wrong, this code is used by your app, it’s part of your app, pretending it’s not will get you in to trouble. You depend on other people’s code and they are just as likely to write new bugs as you are, probably more so. Checking all of that code in to source control gives you a way to audit every line that ever changed in your application. It allows you to use $ git bisect locally and be ensured that it’s the same as in production and that every machine in production is identical. No more tracking down unknown changes in dependencies, all the changes, in every line, are viewable in source control.
In summary, the question is this: Why would one gitignore (i.e. not version control) node_modules but not do the same for Composer's vendor/ directory?
The reason to commit external dependencies is
it's easier to deploy with git pull
the code used is directly included in the commit anyone checks out
Arguments against this are
Git is no deployment tool
all dependency managers do have a way to make exactly sure the code used can be fetched
I don't know anything about npm, but for Composer that last point is implemented by committing composer.lock.
I don't think the "audit code" argument is a valid one in every case. If you do write software that get's audited by itself, and subsequently needs all libraries to be audited, then probably all code changes need to be conserved. This isn't true for the general case.
git bisect works still as well with a committed composer.lock. It does require installing the dependencies with every bisect step, but this is just one simple step, probably already done in the automatic test suite run.
The last thing to worry about is when used packages go offline. This really is more of a problem with Composer, because there is no central hosting of the downloadable releases (npm probably does this). If this is a problem, either commit the code (and try to figure out how to update that missing package in the future), or setup an instance of Satis to create a local copy of the code you use.
Putting all your modules in you VCS makes it really heavy to download or upload. For example, I work on two node.js projects and the total node_modules directories size is between 250MB and 500MB whereas the whole code with assets is generally less than 40MB. Every Node.js developer likes Node lightness, so the code must stay easy to download and share.
For the second point, an alternative to avoid regressions is to be more restrictive in your package.json dependencies versions. You will find more information here.
Finally the best argument is to take a look on the famous modules everybody know :
express ignores
node_modules
mocha ignores
node_modules
q ignores node_modules
...
The more I research this the more I'm starting to think that there is no one correct answer to this, just different opinions as well as pros and cons of each method.
This blog, looking from the context of Bower, does a good job of weighing the pros and cons of each: http://addyosmani.com/blog/checking-in-front-end-dependencies/.
In short: At least for right now, weigh the pros and cons and decide what best fits your situation.
I am in the process of introducing NuGet into our software dev process, both for external binaries (eg Moq, NUnit) and for internal library projects containing shared functionality.
TeamCity is producing NuGet packages from our internal library projects, and publishing them to a local repository. My modified solution files use the local repository for accessing the NuGet packages.
Consider the following source code solutions:
Company.Interfaces.sln builds Company.Interfaces.1.2.3.7654.nupkg.
Company.Common.sln contains a reference to Company.Interfaces via its NuGet package, and builds Company.Common.1.1.1.7655.nupkg, with Company.Interfaces.1.2.3.7654 included as a dependency.
The Company.DataAccess.sln uses the Company.Common nupkg to add
Company.Interfaces and Company.Common as references. It builds
Company.DataAccess.1.0.8.7660.nupkg, including Company.Common.1.1.1.7655 as a dependent component.
Company.Product.A is a website solution that contains references to all three library projects (added by selecting the
Company.DataAccess NuGet package).
Questions:
If there is a source code change to Company.Interfaces, do I always need to renumber and rebuild the intermediate packages (Company.Common and Company.DataAccess) and update the packages in Company.Product.A?
Or does that depend on whether the source code change was
a bug fix, or
a new feature, or
a breaking change?
In reality, I have 8 levels of dependent library packages. Is there tooling support for updating an entire tree of packages, should that be necessary?
I know about Semantic Versioning.
We are using VS2012, C#4.0, TeamCity 7.1.5.
It is a good idea to update everything on each check-in, in order to test it early.
What you're describing can be easily managed using artifact dependencies (http://confluence.jetbrains.com/display/TCD7/Artifact+Dependencies) and "Finish Build" build triggers (or even solely "Nuget Dependency Trigger").
We wrote our own build configuration on the base project (would be Company.Interfaces.sln in this case) which builds and updates the whole tree in one go. It checks in updated packages.config files and .nuspec files along the way. I can't say how much of a time-saver this ended up being for us, even if it might sound like overkill at the beginning.
One thing to watch out for: the script we wrote checks in the files even if the chain fails somewhere in between, to give us the chance of fixing it on our local machine, check in the fix and restart the publishing.