Errors under Jest in CRA using Yarn portal - jestjs

I have a Create-React-App 5 project using Yarn 3 and TypeScript 4. The web app uses a library in the same repo through Yarn's portal: protocol (Yarn workspaces are not used).
myRepo
├── myLib
│   ├── package.json
│   ├── yarn.lock
│   └── src
│    ├── index.js
│   └── macro.js
├── myApp
│   ├── package.json
│   ├── yarn.lock
│   └── src
│   ├── App.tsx
│   ├── App.test.tsx
│   └── index.tsx
└── yarnrc.yml
myApp/package.json uses a portal like so:
"dependencies": {
"myLib": "portal:../myLib",
My top-level Yarn config:
# yarnrc.yml
nodeLinker: node-modules
If I substitute file: for portal:, everything works fine. Even with portal:, yarn build and yarn start within myApp still work fine, but yarn test becomes a problem: /Users/luke/myRepo/myLib/src/index.js: The macro imported from "./macro" must be wrapped in "createMacro" which you can get from "babel-plugin-macros".
Huh? I'm certainly not trying to do anything with Babel macros. Moreover, that stuff is in a library and not the app itself.
What's going on, and how do I fix it?

Okay, I think I've figured it out!
First, CRA includes babel-plugin-macros (source), which operates by looking for particular filenames such as macro.js and treating them as Babel macros (docs). So, if I had a file myApp/src/macro.js, it would error out in a similar way, but also for build/start.
Second, why is it only a problem under Jest? Jest is actually transpiling the dependency, which Webpack somehow knows not to do. Normally this would be inconsequential (apart from slowing things down), but in this case Jest's transpiling is what runs into the problem, since it wrongly tries to interpret myLib using Babel macros (which myLib's author of course never intended).
Third, why is Jest trying to transpile that other library, which lies outside myApp, in the first place? It certainly shouldn't be. Actually, Jest tries to transpile everything besides what is explicitly excluded, and the exclusion under CRA defaults to (source):
transformIgnorePatterns: [
'[/\\\\]node_modules[/\\\\].+\\.(js|jsx|mjs|cjs|ts|tsx)$',
'^.+\\.module\\.(css|sass|scss)$',
],
The intent is to include the app itself and exclude its dependencies. This approach sort of assumes that all the dependencies of myApp will be somewhere under a node_modules, which (PnP aside) would be an especially reasonable expectation under nodeLinker: node-modules. But with a portal, there is a symbolic link to myLib, which is (after resolving its real path) not under any node_modules and therefore does not match the ignore patterns. Thus, dependencies accessed through portals are not excluded from Jest transformation under this config.
Solution
To solve it, configure Jest to also exclude anything not under the project's root directory, i.e., add:
'^(?!<rootDir>)'
Since I happen to be using Craco already, I can just:
// craco.config.js
module.exports = {
jest: {
configure: jestConfig => ({
...jestConfig,
transformIgnorePatterns: [
...jestConfig.transformIgnorePatterns,
'^(?!<rootDir>)',
],
}),
},
};
There really ought to be a built-in way to handle this, but 🤷‍♂️

Related

`Cannot find module` for my own TypeScript module

I have just published a new TypeScript-based module to the NPM registry, ooafs. However, when I try to install it and import it in another TypeScript project, VSCode gives me the following error on the import statement: Cannot find module 'ooafs'.ts(2307).
This module's source files are compiled to JavaScript to a dist/ folder and definitions (.d.ts) are also generated.
Here's the tree of the published module (the one we download when we npm install):
.
├── dist
│ ├── Entry.d.ts
│ ├── EntryFilter.d.ts
│ ├── EntryFilter.js
│ ├── Entry.js
│ ├── EntryType.d.ts
│ ├── EntryType.js
│ ├── FSTypings.d.ts
│ ├── FSTypings.js
│ ├── index.d.ts
│ └── index.js
├── LICENSE
├── package.json
└── README.md
The package.json does contain the following entries:
{
"main": "dist/index.js",
"types": "dist/index.d.ts",
...
}
Because the module works normally on Runkit (pure JS), I assume the only problem I have is related to TypeScript, and it's not the first time TypeScript tells me a module doesn't exist when missing declaration files are the only problem.
Am I missing a step in the compilation process ?
Are my package.json properties wrong ?
If you need to see more code, the Github link is at the beginning of the question, and the published module structure can be found here: https://unpkg.com/ooafs#0.1.2/dist/.
Actually, the problem didn't come from my module (ooafs). It was a problem with the tsconfig.json of the project I was using the module in: The module property must be set to commonjs apparently.
Very late edit:
Also, I highly recommend setting esModuleInterop to true which allows you to import non-es6 modules in a more natural manner.
The answer is not the fix, and is certainly not ideal when you have to use top-level awaits (which don't work on commonjs).
You want to make sure your import path is the final file that node will try and load. So you cannot rely on folders resolving to folder/index.js and you cannot rely on giving file names without extensions (give the ".js" extension)

How to build a docker image from a nodejs project in a monorepo with yarn workspaces

We are currently looking into CI/CD with our team for our website. We recently also adapted to a monorepo structure as this keeps our dependencies and overview a lot easier. Currently testing etc is ready for the CI but I'm now onto the deployment. I would like to create docker images of the needed packages.
Things I considered:
1) Pull the full monorepo into the docker project but running a yarn install in our project results in a total project size of about 700MB and this mainly due to our react native app which shouldn't even have a docker image. Also this should result in a long image pull time every time we have to deploy a new release
2) Bundle my projects in some kind of way. With our frontend we have working setup so that should be ok. But I just tried to add webpack to our
express api and ended up with an error inside my bundle due to this issue: https://github.com/mapbox/node-pre-gyp/issues/308
3) I tried running yarn install only inside the needed project but this will still install my node_modules for all my projects.
4) Run the npm package: pkg. This results in a single file ready to run on a certain system with a certain node version. This DOES work but I'm not sure how well this will handle errors and crashes.
5) Another solution could be copying the project out of the workspace and running a yarn install on it over there. The issue with this is that the use of yarn workspaces (implicitly linked dependencies) is as good as gone. I would have to add my other workspace dependencies explicitly. A possibility is referencing them from a certain commit hash, which I'm going to test right now. (EDIT: you can't reference a subdirectory as a yarn package it seems)
6) ???
I'd like to know if I'm missing an option to have only the needed node_modules for a certain project so I can keep my docker images small.
I've worked on a project following a structure similar to yours, it was looking like:
project
├── package.json
├── packages
│   ├── package1
│  │  ├── package.json
│  │ └── src
│   ├── package2
│  │  ├── package.json
│  │ └── src
│   └── package3
│    ├── package.json
│  └── src
├── services
│   ├── service1
│  │  ├── Dockerfile
│  │  ├── package.json
│  │ └── src
│   └── service2
│    ├── Dockerfile
│    ├── package.json
│  └── src
└── yarn.lock
The services/ folder contains one service per sub-folder. Every service is written in node.js and has its own package.json and Dockerfile.
They are typically web server or REST API based on Express.
The packages/ folder contains all the packages that are not services, typically internal libraries.
A service can depend on one or more package, but not on another service.
A package can depend on another package, but not on a service.
The main package.json (the one at the project root folder) only contains some devDependencies, such as eslint, the test runner etc.
An individual Dockerfile looks like this, assuming service1 depends on both package1 & package3:
FROM node:8.12.0-alpine AS base
WORKDIR /project
FROM base AS dependencies
# We only copy the dependencies we need
COPY packages/package1 packages/package1
COPY packages/package3 packages/package3
COPY services/services1 services/services1
# The global package.json only contains build dependencies
COPY package.json .
COPY yarn.lock .
RUN yarn install --production --pure-lockfile --non-interactive --cache-folder ./ycache; rm -rf ./ycache
The actual Dockerfiles I used were more complicated, as they had to build the sub-packages, run the tests etc. But you should get the idea with this sample.
As you can see the trick was to only copy the packages that are needed for a specific service.
The yarn.lock file contains a list of package#version with the exact version and dependencies resolved. To copy it without all the sub-packages is not a problem, yarn will use the version resolved there when installing the dependencies of the included packages.
In your case the react-native project will never be part of any Dockerfile, as it is the dependency of none of the services, thus saving a lot of space.
For sake of conciseness, I omitted a lot of details in that answer, feel free to ask for precision in the comment if something isn't really clear.
After a lot of trial and error I've found that using that careful use of the file .dockerignore is a great way to control your final image. This works great when running under a monorepo to exclude "other" packages.
For each package, we have a similar named dockerignore file that replaces the live .dockerignore file just before the build.
e.g.,
cp admin.dockerignore .dockerignore
Below is an example of admin.dockerignore. Note the * at the top of that file that means "ignore everything". The ! prefix means "don't ignore", i.e., retain. The combination means ignore everything except for the specified files.
*
# Build specific keep
!packages/admin
# Common Keep
!*.json
!yarn.lock
!.yarnrc
!packages/common
**/.circleci
**/.editorconfig
**/.dockerignore
**/.git
**/.DS_Store
**/.vscode
**/node_modules
I have a very similar setup to Anthony Garcia-Labiad on my project and managed to get it all up&running with skaffold, which allows me to specify the context and the docker file, something like this:
apiVersion: skaffold/v2beta22
kind: Config
metadata:
name: project
deploy:
kubectl:
manifests:
- infra/k8s/*
build:
local:
push: false
artifacts:
- image: project/service1
context: services
sync:
manual:
- src: "services/service1/src/**/*.(ts|js)"
dest: "./services/service1"
- src: "packages/package1/**/*.(ts|js)"
dest: "./packages/package1"
docker:
dockerfile: "services/service1/Dockerfile"
We put our backend services to a monorepo recently and this was one of a few points that we had to solve. Yarn doesn't have anything that would help us in this regard so we had to look elsewhere.
First we tried #zeit/ncc, there were some issues but eventually we managed to get the final builds. It produces one big file that includes all your code and also all your dependencies code. It looked great. I had to copy to the docker image only a few files (js, source maps, static assets). Images were much much smaller and the app worked. BUT the runtime memory consumption grew a lot. Instead of ~70MB the running container consumed ~250MB. Not sure if we did something wrong but I haven't found any solution and there's only one issue mentioning this. I guess Node.js load parses and loads all the code from the bundle even though most of it is never used.
All we needed is to separate each of the packages production dependencies to build a slim docker image. It seems it's not so simple to do but we found a tool after all.
We're now using fleggal/monopack. It bundles our code with Webpack and transpile it Babel. So it produces also one file bundle but it doesn't contain all the dependencies, just our code. This step is something we don't really needed but we don't mind it's there. For us the important part is - Monopack copies only the package's production dependency tree to the dist/bundled node_modules. That's exactly what we needed. Docker images now have 100MB-150MB instead of 700MB.
There's one easier way. If you have only a few really big npm modules in your node_modules you can use nohoist in your root package.json. That way yarn keeps these modules in package's local node_modules and it doesn't have to be copied to Docker images of all other services.
eg.:
"nohoist": [
"**/puppeteer",
"**/puppeteer/**",
"**/aws-sdk",
"**/aws-sdk/**"
]

Project Setup: Creating a Typescript library package for nodejs using npm and webpack

I want to create a library in Typescript that I can share via npm. Specifically, I want to use webpack to generate a js bundle along with a definition file to share the types with the js. So I'd have a tree of files like:
├── lib
│   ├── lib.d.ts
│   └── lib.min.js
├── test
...
├── ts
│   ├── errors
│   │   ├── CannotModifyAlteredObject.ts
│   ├── Lib.ts
│   ├── PostProcessors.ts
│   ├── Serializers.ts
├── tsconfig.json
├── typings.json
├── LICENSE
├── package.json
├── README.md
└── webpack.lib.config.js
And all the types exported by ts/Lib.ts would be exported to a single .d.ts in the lib directory to sit next to the js bundle.
I've looked at the following questions/sources:
Writing npm modules in typescript
How to create a typescript library (and the question it duplicates)
This unanswered question
The offical typescript guide to creating packages
This example typescript library project
And another SO question
However, none of these provide an example using webpack. Being able to bundle everything you need to use the library (apart from the nodejs runtime) into a single file is pretty important for my use case, so webpack fits this role well. I'd like to be able to generate a .d.ts file that maps to what webpack creates. However, I want to avoid creating the .d.ts file manually - it should be possible to automatically extract the types without having manually created .d.ts files get out of sync with my source code. Is there a way of doing this?

How to Target ES6 for Server and ES5 for Client

I am creating a project with node on the server and Angular 2 on the client. I want to use Typescript across the entire project. I would like to target ES6 for the server, since node supports it, but ES5 for the client. Currently my directory structure looks something like this.
├── app.js
├── something.js
├── tsconfig.json
├── wwwroot
│ ├── index.html
│ ├── main.ts
│ ├── components
│ │ ├── mycomponent.ts
I want everything above wwwroot to target ES6, but everything inside of wwwroot to target ES5. I tried putting a second tsconfig.json inside the wwwroot folder, but that didn't seem to work. I am using Atom's autocompile feature, if that is of any relevance.
You can use babel to transpile es6 javascript to whatever flavor of ecmascript you desire.
http://babeljs.io/
I have a similar situation, and what I'm doing is:
- project-root
- scripts
- tsconfig.json
- wwwroot
- scripts
- tsconfig.json
- html
- index.html
- styles
- index.css
The scripts/tsconfig.json targets es6 for the server, and the wwwroot/scripts/tsconfig.json targets es5 for the client.
You can use Babel to transpile to es5 what nodejs does not already implement, using the preset "babel-preset-es2015-node6".
You can transpile only the directories that you want to es5 using for example a script in your node package.json:
"prestart": "babel server-source --out-dir dist/source"

Bower and Sass in a Rails-like folder structure

So I'm working on this project that has a Rails-like folder structure though it's handled by Node.js tooling (Grunt as task runner). I'm using Bower to manage my vendor assets.
My folder structure looks like this:
.
└── src
├── app
│   └── assets
│   ├── javascripts
│   └── stylesheets
│   └── application.scss
├── public
└── vendor
└── bower
Basically all the development source code lives in the app/assets folder, public is where production files go and vendor is where 3rd party stuff goes.
So as you can see, I have this application.scss file. This is the stylesheet manifest I'm using. It's responsible to import all the modules that should be compiled to my final stylesheet file later.
The problem is that I don't see a sane way to reference libraries installed through Bower from inside my manifest file.
With Rails Asset Pipeline/Sprockets I would do //= require_tree vendor/bower and that would work but I don't know what's the equivalent of doing that on the context of this project.
Do you guys have any suggestion on what could I do?
Ps.: Using Grunt tasks to "handle" this is out of question.
Just configure Bower to install packages on vendor/assets/components by creating a file called .bowerrc in your root directory.
{"directory": "vendor/assets/components"}
Everything inside vendor/assets and app/assets is added to the load path, so you can just reference those files.
You may need to put the actual file you want to load. Let's say you installed the normalize-scss package. You'll probably have to add this to your application.scss file:
#import "normalize-scss/normalize";
This is just a guess, but I'd bet on it.
EDIT: This will work on Rails apps, which apparently isn't your case. So if you're using Grunt to compile SCSS, you can add the Bower directory to your load path with the loadPath option.
The Gruntfile's Sass task may look with something like this:
{
sass: {
dist: {
files: {"src/assets/stylesheets/application.scss": "public/application.css"},
options: {
loadPath: ["vendor/bower"]
}
}
}
}
To import the file, you will do something like I said above (referencing the whole file). Didn't test the Grunt configuration, but it'll probably work.
Bower downloads whole git repositories.
Example:
bower install jquery
This would create the following structure
tree vendor/bower
bower
└── jquery
├── bower.json
├── component.json
├── composer.json
├── jquery.js
├── jquery-migrate.js
├── jquery-migrate.min.js
├── jquery.min.js
├── jquery.min.map
├── package.json
└── README.md
1 directory, 10 files
It doesn't make much sense to load all those files in my opinion.
What You could do is:
create a vendor/require directory
symlink all required files into this directory:
cd vendor/require; ln -s ../bower/jquery/jquery.min.js
then require all files with ruby's help or manually
Dir['path/to/vendor/require/*.js'].each do |file_name|
puts "<script type="text/javascript" src="#{file_name}"></script>"
end
You could also use Grunt and it's concat task:
grunt.initConfig({
concat: {
options: {
separator: ';',
},
dist: {
src: ['path/to/vendor/bower/jquery/jquery.min.js', 'path/to/vendor/bower/other-package/init.min.js'],
// or if You decide to create those symlinks
// src: ['path/to/vendor/require/*'],
dest: 'path/to/public/js/built.js',
},
},
});
For compass on sass You could use:
#import 'path/to/directory/*';

Resources