The problem is that let's say I want to start multiple services (several npm start) concurrently, it would be inconvenient to run the services as separate projects. I want to have a folder structure similar to the following under one project workspace:
project
├── service1
│ ├── node_modules
│ │ ├── #module1
│ │ └── #module2
│ ├── package-lock.json
│ ├── package.json
│ ├── public
│ └── src
├── service2
│ ├── node_modules
│ │ ├── #module1
│ │ └── #module2
│ ├── package-lock.json
│ ├── package.json
│ ├── public
│ └── src
└── service3
├── node_modules
│ ├── #module1
│ └── #module2
├── package-lock.json
├── package.json
├── public
└── src
What would be a clean way to do so? I need to start multiple services together and obviously debug any in a convenient fashion if a possibility exists.
You may want to look into Lerna or Nx.
Both are tools that manage mono-repo microservices.
There are some subtle differences between them, but essentially both do the same thing.
They offer ways to share dependencies between your microservices.
They offer ways to created shared libraries.
They offer ways to launch
multiple services together.
Lerna
One of the subtle differences, is that Nx will force you to use a single package.json in your root folder, essentially forcing you to use the same dependencies for all microservices. By contrast, Lerna still allows a specific package.json in each individual folder, which seems to resemble your current directory structure better.
In general, I think Lerna is a safe choice. And you can find a good tutorial here.
Nx
On the other hand, even though Lerna has been around for a longer time it has some quirks at times. I believe Nx is probably technically a more robust solution.
However, I must admit that I've mostly seen it being used for mono-repo front-end projects, and less often for back-ends. Technically, it should be able to handle both.
To get you started with Nx, you could follow this tutorial.
Spoiler: Nx has commands like nx run-many that can help you to execute multiple services together. After migrating to nx, you could then put that command in your "start": script of the package.json, so that npm run start and npm start will execute it.
Related
I may have messed something up with my installation, I don't know.
I reinstalled node with N package manager (script? package?) some time ago to solve sudo problems for my global packages, and today I encountered a problem -- I couldn't require globally installed packages in the node REPL (require('lodash') for example).
I am pretty sure I have something messed up with my node setup, but the more I skim through the web the more confused I am -- what are $NODE_MODULES, $PREFIX, .../lib/node, .../lib/node_modules, $N_PREFIX, which and why are obsolete, what are the differences etc.
My current setup looks like that:
# fragment of ~/.bashrc
# setup NODE
export N_PREFIX="$HOME/.n"
export PATH=$N_PREFIX/bin:$PATH
directory structure:
# output of `tree $N_PREFIX -L 2`. Arrows are symlinks, bin/node is executable
/home/tooster/.n
├── bin
│ ├── check -> ../lib/node_modules/checker/cli.js
│ ├── eslint -> ../lib/node_modules/eslint-cli/bin/eslint.js
│ ├── eslint-cli -> ../lib/node_modules/eslint-cli/bin/eslint.js
│ ├── ffmpeg-bar -> ../lib/node_modules/ffmpeg-progressbar-cli/lib/main.js
│ ├── js-yaml -> ../lib/node_modules/js-yaml/bin/js-yaml.js
│ ├── node
│ ├── npm -> ../lib/node_modules/npm/bin/npm-cli.js
│ ├── npx -> ../lib/node_modules/npm/bin/npx-cli.js
│ ├── tsc -> ../lib/node_modules/typescript/bin/tsc
│ └── tsserver -> ../lib/node_modules/typescript/bin/tsserver
├── include
│ └── node
├── lib
│ └── node_modules
├── n
│ └── versions
└── share
├── doc
├── man
└── systemtap
And the return of 'module' from Node REPL:
Module {
id: '<repl>',
path: '.',
exports: {},
filename: null,
loaded: false,
children: [],
paths: [ <-- None of these actually exist
'/home/tooster/repl/node_modules',
'/home/tooster/node_modules',
'/home/node_modules',
'/node_modules',
'/home/tooster/.node_modules',
'/home/tooster/.node_libraries,
'/home/tooster/.n/lib/node'
]
}
I cannot find any info about difference between the .../lib/node and .../lib/node_modules. Symlinking latter to the former seems to work, but I don't know what are consequences of doing that, so I'd rather not do that blindly.
Also npm config get prefix returns /home/tooster/.n.
How should a proper installation of npm with N look like? I want to have my globally installed packages available in REPL, because I often write quick scripts in Node.
When installing Apache Spark 2.2.1 through Homebrew, the resulting install location seems to have two slightly-different bin/ directories, one a level below the other. (Directory structure at the bottom of this question.)
My main concern is that the load-spark-env.sh (Spark environment variable load script) looks pretty drastically different between the two, and it's tough to confirm which is being used.
In short, I'm wondering:
Why might there be two similar bin/ directories here? Sorry if I'm missing something obvious about Spark setup.
If I have $SPARK_HOME set to libexec/ (see below), will the bin/ here always be referenced over the other directory, or are there other environment variables that I need to set?
Info
I have the following set in ~/.bash_profile:
export JAVA_HOME="/Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home"
export SPARK_HOME="/usr/local/Cellar/apache-spark/2.2.1/libexec"
export PYSPARK_PYTHON="/Users/brad/anaconda3/bin/python3"
Homebrew's install file is here, for reference.
The directory looks like this (I've excluded some irrelevant folders such as sbin):
Cellar$ tree -l apache-spark/
apache-spark/
└── 2.2.1
├── INSTALL_RECEIPT.json
├── LICENSE
├── NOTICE
├── README.md
├── bin
│ ├── find-spark-home
│ ├── load-spark-env.sh
│ ├── pyspark
│ ├── run-example
│ ├── spark-beeline
│ ├── spark-class
│ ├── spark-shell
│ ├── spark-sql
│ ├── spark-submit
│ └── sparkR
└── libexec
├── bin
│ ├── find-spark-home
│ ├── load-spark-env.sh
│ ├── pyspark
│ ├── run-example
│ ├── spark-beeline
│ ├── spark-class
│ ├── spark-shell
│ ├── spark-sql
│ ├── spark-submit
│ └── sparkR
├── conf
├── ...
└── yarn
Edit: I've noticed this same structure with brew-installed Hadoop, so it looks like the question pertains more to Homebrew than the tools it installs.
Why might there be two similar bin/ directories here? Sorry if I'm missing something obvious about Spark setup.
The higher level bin directory contains just some wrappers that override the JAVA_HOME variable and start the executables in the libexec directory. These wrappers are then placed in your /usr/local/bin directory so that you can use things like spark-shell without specifying the full path.
If I have $SPARK_HOME set to libexec/ (see below), will the bin/ here always be referenced over the other directory, or are there other environment variables that I need to set?
If you do need to set the SPARK_HOME directory for whatever reason, then you're right to set it to the libexec one and not the higher level bin directory.
I am currently trying to create a docker container for a node.js project that contains a local dependency. This seems to cause an issue with docker so as a workaround I am trying to just copy the local dependency folders and just ignore their dependency entries in the package.json file. Is there a way to specify dependencies I would like to ignore and have npm install run and skip those enties?
That can be done using devDependencies
The npm modules which you require only to develop, e.g.: unit tests, Coffeescript to Javascript transpilation, minification etc,make the required module a devDependency.
To skip Installation of devDepenencies pass --production flag to npm install,with the --production flag(or NODE_ENV environment variable set to production) npm will not install modules listed in devDependencies."
npm install --production
To make any module to be part of devDependencies pass --dev while installing.
npm install packagename --save-dev
It is a common issue, not only with Docker, but also with some cloud deployments. For example deployment to CloudFoundry using standard Node.js buildpack will cause npm install/yarn to run anyway. So, you'll also need to apply some tricks to work with local modules
If you don't mind to switch from NPM to Yarn for dependency management, you can use workspaces feature.
My package.json looks like this:
{
...
"dependencies": {
"some-module-i-want-to-install": "1.0.0",
"another-module-i-want-to-install": "1.0.0",
"#my/local-dependency-one": "1.0.0",
"#my/local-dependency-two": "1.0.0"
},
"workspaces": ["packages/*"]
}
And my project source layout has the following structure:
.
├── index.js
├── package.json
├── packages
│ ├── local-dependency-one
│ │ ├── index.js
│ │ └── package.json
│ └── local-dependency-two
│ ├── index.js
│ └── package.json
└── yarn.lock
After running yarn, modules I want to install are fetched from NPM registry, and local dependencies are installed from packages directory to node_modules.
.
├── index.js
├── node_modules
│ ├── #my
│ │ ├── local-dependency-one
│ │ │ └── ...
│ │ └── local-dependency-two
│ │ └── ...
│ ├── another-module-i-want-to-install
│ │ └── ...
│ └── some-module-i-want-to-install
│ └── ...
├── package.json
├── packages
│ ├── local-dependency-one
│ │ └── ...
│ └── local-dependency-two
│ └── ...
└── yarn.lock
As you can see, I prefer to define my local packages as scoped (#my/...). It is not mandatory, but a best practice. NPM treats scoped packages as private by default, so I don't need to worry that they will be occasionally published or explicitly mark them as private.
I'm interested in learning how the #shopify/polaris project in github is built and published to npm. My main questions are:
How are the index.es.js and index.js files being generated? Are those being generated programmatically on my computer, or are they published to npm like this?
What mechanism is preventing the files in the github repo from being downloaded when installed? I don't see a .npmignore in the repo.
Below I have the files in the npm package, and the github, and you can see they're different.
Here's what the polaris project looks like when it's installed via NPM / yarn.
.
├── CHANGELOG.md
├── README.md
├── index.es.js
├── index.js
├── package.json
├── src
├── styles
├── styles.css
├── styles.scss
└── types
Here's what the project looks like on github.
.
├── CHANGELOG.md
├── README.md
├── circle.yml
├── config
├── documentation
├── examples
├── package.json
├── scripts
├── src
├── tests
├── tsconfig.json
├── tslint.json
└── yarn.lock
We use Rollup to generate the different entry files for Polaris. You can see our generic config file here: https://github.com/Shopify/polaris/blob/master/config/rollup/index.js. Note that it does all the work of compiling our TypeScript source files (using TypeScript and Babel), as well as kicking off the work required to bundle our CSS and icons as appropriate. This config generator is then run three times, which you can see here: https://github.com/Shopify/polaris/blob/master/scripts/build.js#L36-L38
The main entry file (.js, and spits out the CSS for our CDN)
The module entry file (.es.js)
The embedded entry file
We then copy these into their final locations for the NPM publish.
As for how we control what files are in the NPM package, we use the files key in our package.json; these are the relevant lines.
I have a node.js app using a single git repo, that has various packages. I am trying to make things modularized, and separate.
├── common
│ └── formattr
│ ├── node_modules
│ ├── package.json
│ └── formattr.js
└── providers
├── aws
│ ├── node_modules
│ └── package.json
└── google
├── node_modules
└── package.json
The problem I am running into is that when somebody clones the app, they have to recurse into each package direction and manually run npm install. Additionally, there is lots of wasted resources and duplication, since the package aws might use npm install request but so does the google package. Right now, they both store independ versions.
Finally, there is this problem as well:
// Inside of the aws package, I need formattr. This makes me cry inside.
let format = require('../../common/formattr/formattr');
What is the recommended pattern and structure?