I'm using Aurelia and the new aurelia-webpack-plugin (2.0.0-rc.2). I want to use this as a "multiproject build":
.
├── project1
├── project2
├── project3
├── build
The three projects are all using Aurelia (same versions), but are independent. To avoid build script duplicates I extracted the webpack.config.js and several other scripts to the build folder. The projects are calling them via npm scripts from the package.json of the project. The working directory of this node-process is changed to the build folder before webpack is called.
Some relevant configs in my webpack config:
...
new AureliaPlugin({
includeAll: path.resolve(`${projectDir}/app/`),
viewsFor: `${path.resolve(projectDir)}/app/**/*.{ts,js}`
})
...
entry: {
main: "aurelia-bootstrapper"
},
resolve: {
modules: [`${projectDir}/app`, `${projectDir}/node_modules`, "node_modules"]
}
...
projectDir points to the specific root directory of the project.
The problem is now, that the GlobDependenciesPlugin inside the aurelia-webpack-plugin does not find the entrypoints (Main.ts) of the projects.
After debugging I found at least two reasons for this:
this.root in GlobDependenciesPlugin points to the current cwd of the node process, which is the build folder (there is no way to configure the value)
only the last "node_modules" folder in the modules array from the config is used for searching. The others are filtered out, cause of relative paths.
Is there a way to get this working with this shared build script structure?
Related
I have recently packaged an electron app using electron-builder:
myProject/
├── package.json
├── app/
└── release/
All files created by electron-builder are place in the release directory. The executable works fine on my local machine, with all features present through the packaged app.
However, once I move the application to another machine only some features are available. Noticeably features within subdirectories in app/ are not included.
For example here a snippet of the app/ directory:
app/
├── app.html
├── index.js
├── components/
└── other files and folders
Features added from .js/.html files within components/ are not present when I move the app to another machine. I have tried both moving just the executable as well as the whole release/ directory, neither includes additional features beyond what is included in app.html.
Update
It does indeed look like any other machine simply doesn't read items contained in
<script></script>
In my app.html file
Would there be some outside installation I need to do on another machine to get this executable running
Found the issue,
It involved my usage of a two package.json structure
Both dependencies and devDependencies of my build were located in the root/package.json, where dependencies needed to be moved to the app/package.json file
I have the following situation:
Frontend's package.json
{
"dependencies": {
"lib" : "1.0.0",
"foo" : "1.0.0"
}
}
lib's package.json
{
"devDependencies": {
"foo" : "1.0.0"
},
peerDependencies": {
"foo" : "1.0.0"
}
}
While I am developing with webpack-dev-server some Frontend I am linking the module foo with a npm link leading into this dependency tree of the Frontend:
├── foo#1.0.0
└─┬ lib#1.0.0
└── foo#1.0.0
Rather than having something like this:
├── foo#1.0.0
└── lib#1.0.0
I have already found out that webpack finds the module foo in my lib in its node_modules folder, because I defined this package as devDependencies for my unit tests. Now I have the package twice in my bundled source which leads into some bugs.
Question: How can I force webpack or npm to use the module foo of my parent (the Frontend) in my lib like the peerDependencies suggests?
I assume you have frontend and lib checked out locally in two different folders and linked lib to frontend using something like cd frontend && npm link lib. So you goal is to work (change code in two editors) on both packages at the same time, but also want to be able to run your compiler and unit tests in frontend and lib separately.
The answer is: you can't (that easy). That's because dependencies are resolved always first to the folder of the package first. That means, if you have foo installed in lib, it will always pick lib:foo, not root:foo.
The simplest solution for that is to not install lib:foo at all, or just remove it. It's using then root:foo, as you want. If you use npm link however, it installs first all lib's dependencies.
So for a one time build in frontend, you can simply remove that lib:foo package and then build. However, from the point of view of the lib package nothing will work as its dependencies got screwed.
The more robust solution is to install lib in root in a away that only installs the actual dependencies of foo that are not in peer/devDependencies
To do so, you can use npm-local-development at https://github.com/marcj/npm-local-development
It basically does the same thing as npm link, but works around the devDependency limitation by setting up a file watcher and syncs file changes automatically in the background, excluding all devDependencies/peerDependencies.
You install npm-local-development: npm i -g npm-local-development
You create file called .links.json in your root package.
You write every package name with its local relative folder path into it like so
{
"#shared/core": "../../my-library-repo/packages/core"
}
Open a console and run npm-local-development in that root package. Let it run in the background.
Disclaimer: I'm the author of this free open-source project.
We are currently looking into CI/CD with our team for our website. We recently also adapted to a monorepo structure as this keeps our dependencies and overview a lot easier. Currently testing etc is ready for the CI but I'm now onto the deployment. I would like to create docker images of the needed packages.
Things I considered:
1) Pull the full monorepo into the docker project but running a yarn install in our project results in a total project size of about 700MB and this mainly due to our react native app which shouldn't even have a docker image. Also this should result in a long image pull time every time we have to deploy a new release
2) Bundle my projects in some kind of way. With our frontend we have working setup so that should be ok. But I just tried to add webpack to our
express api and ended up with an error inside my bundle due to this issue: https://github.com/mapbox/node-pre-gyp/issues/308
3) I tried running yarn install only inside the needed project but this will still install my node_modules for all my projects.
4) Run the npm package: pkg. This results in a single file ready to run on a certain system with a certain node version. This DOES work but I'm not sure how well this will handle errors and crashes.
5) Another solution could be copying the project out of the workspace and running a yarn install on it over there. The issue with this is that the use of yarn workspaces (implicitly linked dependencies) is as good as gone. I would have to add my other workspace dependencies explicitly. A possibility is referencing them from a certain commit hash, which I'm going to test right now. (EDIT: you can't reference a subdirectory as a yarn package it seems)
6) ???
I'd like to know if I'm missing an option to have only the needed node_modules for a certain project so I can keep my docker images small.
I've worked on a project following a structure similar to yours, it was looking like:
project
├── package.json
├── packages
│ ├── package1
│ │ ├── package.json
│ │ └── src
│ ├── package2
│ │ ├── package.json
│ │ └── src
│ └── package3
│ ├── package.json
│ └── src
├── services
│ ├── service1
│ │ ├── Dockerfile
│ │ ├── package.json
│ │ └── src
│ └── service2
│ ├── Dockerfile
│ ├── package.json
│ └── src
└── yarn.lock
The services/ folder contains one service per sub-folder. Every service is written in node.js and has its own package.json and Dockerfile.
They are typically web server or REST API based on Express.
The packages/ folder contains all the packages that are not services, typically internal libraries.
A service can depend on one or more package, but not on another service.
A package can depend on another package, but not on a service.
The main package.json (the one at the project root folder) only contains some devDependencies, such as eslint, the test runner etc.
An individual Dockerfile looks like this, assuming service1 depends on both package1 & package3:
FROM node:8.12.0-alpine AS base
WORKDIR /project
FROM base AS dependencies
# We only copy the dependencies we need
COPY packages/package1 packages/package1
COPY packages/package3 packages/package3
COPY services/services1 services/services1
# The global package.json only contains build dependencies
COPY package.json .
COPY yarn.lock .
RUN yarn install --production --pure-lockfile --non-interactive --cache-folder ./ycache; rm -rf ./ycache
The actual Dockerfiles I used were more complicated, as they had to build the sub-packages, run the tests etc. But you should get the idea with this sample.
As you can see the trick was to only copy the packages that are needed for a specific service.
The yarn.lock file contains a list of package#version with the exact version and dependencies resolved. To copy it without all the sub-packages is not a problem, yarn will use the version resolved there when installing the dependencies of the included packages.
In your case the react-native project will never be part of any Dockerfile, as it is the dependency of none of the services, thus saving a lot of space.
For sake of conciseness, I omitted a lot of details in that answer, feel free to ask for precision in the comment if something isn't really clear.
After a lot of trial and error I've found that using that careful use of the file .dockerignore is a great way to control your final image. This works great when running under a monorepo to exclude "other" packages.
For each package, we have a similar named dockerignore file that replaces the live .dockerignore file just before the build.
e.g.,
cp admin.dockerignore .dockerignore
Below is an example of admin.dockerignore. Note the * at the top of that file that means "ignore everything". The ! prefix means "don't ignore", i.e., retain. The combination means ignore everything except for the specified files.
*
# Build specific keep
!packages/admin
# Common Keep
!*.json
!yarn.lock
!.yarnrc
!packages/common
**/.circleci
**/.editorconfig
**/.dockerignore
**/.git
**/.DS_Store
**/.vscode
**/node_modules
I have a very similar setup to Anthony Garcia-Labiad on my project and managed to get it all up&running with skaffold, which allows me to specify the context and the docker file, something like this:
apiVersion: skaffold/v2beta22
kind: Config
metadata:
name: project
deploy:
kubectl:
manifests:
- infra/k8s/*
build:
local:
push: false
artifacts:
- image: project/service1
context: services
sync:
manual:
- src: "services/service1/src/**/*.(ts|js)"
dest: "./services/service1"
- src: "packages/package1/**/*.(ts|js)"
dest: "./packages/package1"
docker:
dockerfile: "services/service1/Dockerfile"
We put our backend services to a monorepo recently and this was one of a few points that we had to solve. Yarn doesn't have anything that would help us in this regard so we had to look elsewhere.
First we tried #zeit/ncc, there were some issues but eventually we managed to get the final builds. It produces one big file that includes all your code and also all your dependencies code. It looked great. I had to copy to the docker image only a few files (js, source maps, static assets). Images were much much smaller and the app worked. BUT the runtime memory consumption grew a lot. Instead of ~70MB the running container consumed ~250MB. Not sure if we did something wrong but I haven't found any solution and there's only one issue mentioning this. I guess Node.js load parses and loads all the code from the bundle even though most of it is never used.
All we needed is to separate each of the packages production dependencies to build a slim docker image. It seems it's not so simple to do but we found a tool after all.
We're now using fleggal/monopack. It bundles our code with Webpack and transpile it Babel. So it produces also one file bundle but it doesn't contain all the dependencies, just our code. This step is something we don't really needed but we don't mind it's there. For us the important part is - Monopack copies only the package's production dependency tree to the dist/bundled node_modules. That's exactly what we needed. Docker images now have 100MB-150MB instead of 700MB.
There's one easier way. If you have only a few really big npm modules in your node_modules you can use nohoist in your root package.json. That way yarn keeps these modules in package's local node_modules and it doesn't have to be copied to Docker images of all other services.
eg.:
"nohoist": [
"**/puppeteer",
"**/puppeteer/**",
"**/aws-sdk",
"**/aws-sdk/**"
]
I'm relatively new to modern JS development and I need help or advice about this situation I'm in.
Situation: We have a React-Typescript-Redux project supporting IE8 (React 0.14). Now we're upgrading to IE11 and React 16 but IE8 should be supported.
Requirement: Reduce project maintenance between browser versions by using different packages and/ or config files for each build.
Problem: From research I made so far it seems impossible to use different package.json files and node_modules folders inside the same project with selected tools: npm, Webpack, React, Redux, Typescript. Yarn seems to support multiple package.json files but we'd like to avoid migrating from npm if possible.
Current project structure:
project_root/
node_modules/
src/
components/
utils/
index.tsx
components.css
index.html
package.json
tsconfig.json
webpack.config.json
What I thought might work was to introduce IE8 subfolder with its package.json and node_modules folder and then reference that folder for the build task somehow but now I'm oblivious how to tell npm to reference it on build.
Proposed project structure:
project_root/
ie8/
node_modules/
package.json
node_modules/
src/
components/
utils/
index.tsx
components.css
index.html
package.json
tsconfig.json
webpack.config.json
I tried different things found on web, including resolve.modules: [__dirname + "/ie8/node_modules"] but it seems it doesn't work or I misunderstand what it does because I get Cannot find name 'require' errors on several files and Typescript 2.8.3 is referenced in terminal output instead 2.3.4. Without it, project builds with configuration for IE11.
So, can anybody tell me with certainty it's not possible or offer some guidance? This is the closest answer I found so far but doesn't sound final. Alternatively, can project structure like this support what is required or separating project into two is the best bet?
Thanks in advance.
OK, so after some more research I stumbled upon Lerna which mostly allows me to do what I wanted (from what I've seen so far). It requires specific project tree setup, like this:
project_root/
node_modules/
packages/
components/ // Components shared between projects
components/
MyComponent.jsx
index.jsx
legacy/
output/
build.js // React 0.14 build
node_modules/
package.json // project specific dependencies
index.jsx // project specific entry
.babelrc
modern/
output/
build.js // React 16 build
node_modules/
package.json // project specific dependencies
index.jsx // project specific entry
.babelrc
package.json // contains devDependencies shared between projects
lerna.json
webpack.config.js
index.html
Then, in components/index.jsx I specified require commands for different versions based on global variable:
if(PROJECT_SRC == "legacy"){
React = require('../legacy/node_modules/react');
ReactDOM = require('../legacy/node_modules/react-dom');
} else {
React = require('../modern/node_modules/react');
ReactDOM = require('../modern/node_modules/react-dom');
}
Note: This is probably bad practice but the only way at the moment I could include different React versions in the build. I'll have to see what problems arise with this approach after the whole project changes to this model.
In webpack.config.js I configured two exports - one for modern and one for legacy. Each points to a different entry index.jsx file, uses webpack.DefinePlugin to set global variable to "legacy" or "modern", and specifies path to common components module to resolve: ['node_modules', path.resolve(__dirname, 'components')]
webpack.config for a single project output looks something like this:
{
entry: "./packages/legacy/index.jsx",
target: "web",
output:
{
filename: "build.js",
path: __dirname + "/packages/legacy/dist/",
libraryTarget: "var",
library: "lib_name"
},
devtool: "source-map",
resolve: {
extensions: [".js", ".jsx", ".json"],
modules: ['node_modules', path.resolve(__dirname, 'components')]
},
plugins: plugins_legacy,
module: {
loaders: [
{
test: /\.jsx?$/,
loader: "babel-loader",
exclude: /node_modules/
}
]
}
}
Feel free to comment or point to problems but I hope this will help somebody in the future! :)
Using electron in combination with Angular2, Typescript and Electron I am try to find out how to use a node module package installed via npm.
The current problem is that I have to specify the location of the module like var module = require('./node_modules/xyz/lib/xyz.js'). But then electron does not find the dependencies of xyz, which are located within ./node_modules/xyz/node_modules/yyy and complains ./yyy.js can not be found.
The electron app structure
dist
├── angular2.dev.js
├── config.js
├── index.html
├── main.js
├── node_modules
├── package.json
└── app.js
UPDATE:
A similar question has been asked and my answer would most probably help you here:
If you don't append the path to your app node_modules directory under your app root to the NODE_PATH variable it is not going to work. So you need to do something like this:
export NODE_PATH=/PATH/TO/APP/node_modules
electron /PATH/TO/APP
When exporting NODE_PATH make sure that you provide an absolute path.
If electron cannot find modules when you require them normally, it's a sign that your package.json doesn't contain the modules as dependency even if the module is already available under your dist directory.
So make sure that you are inside dist directory and use
npm install --save xyz
note the --save flag!
The current problem is that I have to specify the location of the module like var module = require('./node_modules/xyz/lib/xyz.js')
You should be able to do var module = require('xyz'); If you have it locates in the relative path ./node_modules/ .... that you mentioned.
If you didn't include the modules in your package.json I found it easiest to just copy all of them to node_modules in your release. It's something like releases > ARCHITECTURE > resources > node_modules
According to electron's docs, set the nodeIntegration preference to true. Do this by adding
webPreferences: {
nodeIntegration: true
}
to your createWindow() function.