Background
I have an eslint file that I use across multiple Nodejs projects with my team. This file is important and I need to use it in every project, so I end up copying it over and over again.
Problem
This is a terrible idea because If I make a change to the eslint file, I need to manually copy/paste it into all the other projects and sometimes I forget which projects have the updated file and which projects don't.
Objective
My objective here would be to make it automatic. My first idea was to publish the eslint file in NPM, and then manage it via there.
The problem with this approach is that if I do npm install, NPM will place my lint file into the node_modules directory and thus nothing will work.
This file needs to be at the root of the project, side by side with package.json.
Question
How do I share this eslint file across multiple projects automatically?
Answer
Turns out that you can use NPM packages to share ESLint files across multiple projects and teams, which was unknown to me:
http://engineering.invisionapp.com/post/sharing-eslint-across-teams/
The official documentation details how this can be done, and which conventions to follow:
http://eslint.org/docs/developer-guide/shareable-configs#creating-a-shareable-config
Related
Someday I just got curious about node_module in framework or UI libraries such as React. After searching some stuff, I found there should be no changes in node_modules unless the user really needs to, so here's my questions.
Why there shouldn't be changes in node_modules?
Even I change the code, there were no change in result. Why does this happen? Even deleting the file or folder inside node_modules there were no changes. (I thought it should show an error, but it worked Ok...)
When we start the framework (like npm start in React), does NPM downloads the external files for example from Github everytime and places in the DOM? If that's right, the files in node_modules are just readable ones?
Could someone give me an answer?
node_modules are the libraries / packages / modules (whatever name you call) written by the open source community. They can be inter-depending. If you change one of those files without reviewing the impact to their dependent, the execution of code may crash.
However, not every single file or every single line of codes are required for each execution of code. Most of the time, one package can do things way more than what your code truly needed. If your code doesn't depend on the files that you changed, your project can still run happily.
npm start doesn't download files automatically. npm install does. So files in node_modules are not readable only. However, in many case, files in node_modules were ignored from git commit. In server environment, packages are freshly pulled from remote, instead of from your local machine. Therefore your changes to packages would not be deployed unless you explicitly do so.
Technically you can modify the files in node_modules and NOT running npm update forever - not a good commercial practice. Acceptable for personal project, if you are the sole programmer and can fully control when to update packages.
Well, if you change your node module an npm update will eventually overwrite your code and you will lose your functionality possibly even without knowing where the problem is.
I have the same problem that has already been documented on GitHub here. ui-tinymce references a number of dependencies which cannot be reached in my application.
GET http://localhost:8080/jspm_packages/github/tinymce/tinymce-dist#4.3.12/themes/modern/theme.min.js # angular.js:6084
tinymce.js:9426 Failed to load: /jspm_packages/github/tinymce/tinymce-dist#4.3.12/themes/modern/theme.min.js
I am able to use the workaround suggested in the github issue above, which changes the baseURL. This works fine in my development environment. However, when I run jspm bundle-sfx it does not pick up these dependencies and I am left in the same situation without templates or plugins.
What is the best way to address this? Can angular-ui-tinymce be broken down so that the dependent files are available in separate packages? Or should I just use gulp to get around this problem?
I tried using Gulp to concatenate the missing files, however this will not work because by default tinymce still expects the files to be at the relative locations which it uses in its own internal file structure.
I still think it would be helpful for Tinymce to provide separate packages for the most common themes, however I admit that there are a lot of themes and plugins so this would be a fair amount of work.
In the end the simplest thing to was to copy the theme and plugin files into the "correct" relative directories within my own source code. This way I can change the relative baseURL for tinymce and it will be correct when I run it in production as well as development environments.
This way I can run jspm bundle-sfx and it will bundle these files along with everything else. However you may have to import the files explicitly if you do not serve the area statically in your application. For example:
import 'sysadmin/app/tinymce/themes/modern/theme';
I have a repo which consists of several "micro-services" which I upload to AWS's Lambda. In addition I have a few shared libraries that I'd like to package up when sending to AWS.
Therefore my directory structure looks like:
/micro-service-1
/dist
package.json
index.js
/micro-service-2
/dist
package.json
index.js
/shared-component-1
/dist
package.json
component-name-1.js
/shared-component-2
/dist
package.json
component-name-2.js
The basic deployment leverages the handy node-lambda npm module but when I reference a local shared component with a statement like:
var sharedService = require('../../shared-component-1/dist/index');
This works just fine with the node-lambda run command but node-lambda deploy drops this local dependency. Probably makes sense because I'm going below the "root" directory in my dependency so I thought maybe I'd leverage gulp to make this work but I'm pretty darn new to it so I may be doing something dumb. My strategy was to:
Have gulp deploy depend on a local-deps task
the local-deps task would:
npm build --production to a directory
then pipe this directory over to the micro-service under the /local directory
clean up the install in the shared
I would then refer to all shared components like so:
var sharedService = require('local/component-name-1');
Hopefully this makes what I'm trying to achieve. Does this strategy make sense? Is there a simpler way I should be considering? Does anyone have any examples of anything like this in "gulp speak"?
I have an answer to this! :D
TL;DR - Use npm link to link create a symbolic link between your common component and the dependent component.
So, I have a a project with only two modules:
- main-module
- referenced-module
Each of these is a node module. If I cd into referenced-module and run npm link, then cd into main-module and npm link referenced-module, npm will 'install' my referenced-module into my main-module and store it in my node_modules folder. NOTE: When running the second npm link, the name of the project is the one you find in your package.json, not the name of the directory (see npm link documentation, previously linked).
Now, in my main-module all I need to do is var test = require('referenced-module') and I can use that to my hearts content. Be sure to module.exports your code from your referenced-module!
Now, when you zip up main-module to deploy it to AWS Lambda, the links are resolved and the real modules are put in their place! I've tested this and it works, though not with node-lambda yet, though I don't see why this should be a problem (unless it does something different with the package restores).
What's nice about this approach as well is that any changes I make to my referenced-module are automatically picked up by my main-module during development, so I don't have to run any gulp tasks or anything to sync them.
I find this is quite a nice, clean solution and I was able to get it working within a few minutes. If anything I've described above doesn't make any sense (as I've only just discovered this solution myself!), please leave a comment and I'll try and clarify for you.
UPDATE FEB 2016
Depending on your requirements and how large your application is, there may be an interesting alternative that solves this problem even more elegantly than using symlinking. Take a look at Serverless. It's quite a neat way of structuring serverless applications and includes useful features like being able to assign API Gateway endpoints that trigger the Lambda function you are writing. It even allows you to script CloudFormation configurations, so if you have other resources to deploy then you could do so here. Need a 'beta' or 'prod' stage? This can do it for you too. I've been using it for just over a week and while there is a bit of setup to do and things aren't always as clear as you'd like, it is quite flexible and the support community is good!
While using serverless we faced a similar issue, when having the need to share code between AWS Lambdas. Initially we used to duplication the code, across each microservice, but later as always it became difficult to manage.
Since the development done in Windows Environment, using symbolic links was not an option for us.
Then we came up with a solution to use a shared folder to keep the local dependencies and use a custom written gulp task to copy these dependencies across each of the microservice endpoints so that the dependency can be required similar to npm package.
One of the decisions we made is not to keep two places to define the dependencies for microservices, so we used the same package.json to define the local shared dependencies, where gulp task passes this file and copy the shared dependencies accordingly also installing the npm dependencies with a single command.
Later we made the code open source as npm modules serverless-dependency-install and gulp-dependency-install.
To be completely specific:
I am writing a Node.js app that is intended to be a websocket bot for Slack.
A Node project exists that abstracts the majority of the Slack API. (It is NOT an npm module.)
I'm not overly familiar with grunt, etc. but I can get the dependencies to install and utilize all this code by placing my own mybot.js in the root folder of this git clone and running node mybot.js with mybot.js being based on the files in the example folder.
Committing to my own repository, I don't want to commit any of the aforementioned project code -- it's not mine! I do, however, want it as a dependency. Unfortunately, this code by Slack is not an npm module that makes it easy to do. The project has a /bin folder and a /src folder full of coffee script, etc. that grunt builds to .js files.
The Slack project code has its own dependencies. In my way of thinking, those are sub-dependencies for me, or cascading dependencies. My project only depends on whatever the Slack project depends on.
I would like to be able to update my project with updates (manually, or via build) from the git repo of the Slack project as needed.
It seems there must be a way for me to include this project as a dependency, and once built, properly reference it's bin and src folder objects (bin/slack, src/message, client, channel, user, etc.) without committing it to my own repository. Especially great if it could be in a subfolder separate from my own model definitions. In a way, this seems no different to me than including jQuery in my website layout via a CDN. I'm only asking for the jQuery project and depending on my link flavor, I can get a specific version or the latest version, etc.
So, it turns out the comment by Ben pointing me to the npmjs.com slack-client npm module was the help I really needed. I just didn't really know how to ask the right question, I think.
And while I hate to look a gift horse in the mouth, a little more than a link, Ben, would've saved me another three hours, probably. Perhaps: "It is an npm module, not just a project from github." But thank you, even if it took me a while to decipher what you were saying.
What exactly should I put in .npmignore?
Tests? Stuff like .travis.yml, .jshintrc? Anything that isn't needed when running the module (except the readme)?
I can't find any guidance on this.
As you probably found, NPM doesn't really state specifically what should go in there, rather they have a list of ignored-by-default files. Many people don't even use it as everything in your .gitignore is ignored in npm by default if .npmignore doesn't exist. Additionally, many files are already ignored by default regardless of settings and some files are always excluded from being ignored, as outlined in the link above.
There is not much official on what always should be there because it is basically a subset of .gitignore, but from what I gather from using node for 5-ish years, here's what I've come up with.
Note: By production I mean any time where your module is used by someone and not to develop on the module itself.
Pre-release cross-compiled sources
Pros: If you are using a language that cross-compiles into JavaScript, you can precompile before release and not include .coffee files in your package but keep tracking them in your git repository.
Build file leftovers
Pros: People using things like node-gyp might have object files that get generated during a build that never should go into the package.
Cons: This should always go into the .gitignore anyway. You must place these things inside here if you are using a .npmignore file already as it overrides .gitignore from npm's point of view.
Tests
Pros: Less baggage in your production code.
Cons: You cannot run tests on live environments in the slim chance there is a system-specific failure, such as an out of date version of node running that causes a test to fail.
Continuous integration settings/Meta files
Pros: Again, less baggage. Things such as .travis.yml are not required for using, testing, or viewing the code.
Non-readme docs and code examples
Pros: Less baggage. Some people exist in the school-of-thought where if you cannot express at least minimum viable functionality in your Readme, your module is too big.
Cons: People cannot see exhaustive documentation and code examples on their own file system. They would have to visit the repository (which also requires an internet connection).
Github-pages objects
Pros: You certainly don't need to litter your releases with CNAME files or placeholder index.htmls if you use your module serves double-duty as a gh-pages repository as well.
bower.json and friends
Pros: If you decide to build in your dependencies prior to release, you don't need the end-user to install bower then install more things with that. I would, personally, keep that stuff in the package. When I do an npm install, I should only be relying on npm and no other external sources.
Basically, you should ever use it if there is something you wish to keep out of your npm package but checked-in to your module's repo. It's not a long list of items, but npm would rather build in the functionality than having people stuck with irrelevant objects in their package.
I agree with lante's short and syntetic answer and SamT's big answer:
You should not include your tests in your package.
Your package should only contains production runtime files.
That will make your package more straightforward and faster to be dowloaded.
My contribution to those answers:
.npmignore is the blacklist way to achieve package file selection. But in a more practical way, you can whitelist files you need to include in your package using the files field in your package.json:
{
"files": [
"lib/",
"index.js"
]
}
I think that's simpler, future proof and have better semantics ;)
Just to clarify, anytime someone do npm install your-library, npm will download all source files that the package includes. Those files that were included in the .npmignore file in the source code of the package your-library will be excluded when publishing the lib, so users of your-library won't download them.
Know that people installing your library will need just your library running, anything else will be not necessary.
For example, when someone installs a library, its probably that he/she doesn't care about your .travis.yml or your .jshintrc files, or even some images, Grunt files, documentation, etc.
.npmignore could let your npm package to have less files, and faster to be downloaded
Don't include your tests. Oftentimes tests are like 5x the size of the actual codebase. As long as your tests are on Github, etc, that's good enough.
But what you absolutely should do is test your NPM package in its published format. Create some smoke tests that reside in the actual codebase, but are not part of the test suite.
You can read about testing your package after tarballing it, here:
https://github.com/ORESoftware/r2g
How to test an `npm publish` result, without actually publishing to NPM?