I a pnpm and turborepo monorepo with two apps, a Fastify API and a React frontend. Both apps are bootstrapped using modular components (frontend components, Fastify Plugins, etc.) that are also their own separate packages in the monorepo. I'm trying to refactor and put the things that are shared between the main apps and the components into one single helpers package.
.
├── apps/
│ ├── ui
│ └── api
├── modules/
│ ├── api-module-1
│ ├── api-module-2
│ ├── ui-module-1
│ └── ui-module-2
└── packages/
└── helpers/
├── isomorphic-function-01
├── ui-function-01
├── ui-function-02
├── api-function-01
└── api-function-02
The apps have the modules and packages as workspace dependencies. A UI function can be used with any UI module or the main UI app, and any API function can be used with any UI Module or the main API app and there can also be pure functions that can be used both on the browser and in Node.
The problem is if I export something in the helpers package that is browser specific like a React Hook then the API crashes because it has something that won't load in Node. And if I export something that is API specific such as ORM related, then the frontend crashes as it is not able to load it in the browser.
I am using tsup for prod builds and tsx for dev in the API app, and Vite for the frontend.
The exports in the helpers package is done using a barrel file. I'm also using TypeScript and ESM. If I comment out the API function, UI app will run fine and vice versa.
export * from './isomorphic-function-01';
export * from './ui-function-01';
export * from './api-function-01';
I had expected the apps/modules to only use what is imported and not load the entireity of the helpers package's exports. I can split this into 3 helpers packages for UI, API and Isomorphic and get it to work but is there a way I can keep one package and only allow them to import what is required?
Related
I have a Create-React-App 5 project using Yarn 3 and TypeScript 4. The web app uses a library in the same repo through Yarn's portal: protocol (Yarn workspaces are not used).
myRepo
├── myLib
│ ├── package.json
│ ├── yarn.lock
│ └── src
│ ├── index.js
│ └── macro.js
├── myApp
│ ├── package.json
│ ├── yarn.lock
│ └── src
│ ├── App.tsx
│ ├── App.test.tsx
│ └── index.tsx
└── yarnrc.yml
myApp/package.json uses a portal like so:
"dependencies": {
"myLib": "portal:../myLib",
My top-level Yarn config:
# yarnrc.yml
nodeLinker: node-modules
If I substitute file: for portal:, everything works fine. Even with portal:, yarn build and yarn start within myApp still work fine, but yarn test becomes a problem: /Users/luke/myRepo/myLib/src/index.js: The macro imported from "./macro" must be wrapped in "createMacro" which you can get from "babel-plugin-macros".
Huh? I'm certainly not trying to do anything with Babel macros. Moreover, that stuff is in a library and not the app itself.
What's going on, and how do I fix it?
Okay, I think I've figured it out!
First, CRA includes babel-plugin-macros (source), which operates by looking for particular filenames such as macro.js and treating them as Babel macros (docs). So, if I had a file myApp/src/macro.js, it would error out in a similar way, but also for build/start.
Second, why is it only a problem under Jest? Jest is actually transpiling the dependency, which Webpack somehow knows not to do. Normally this would be inconsequential (apart from slowing things down), but in this case Jest's transpiling is what runs into the problem, since it wrongly tries to interpret myLib using Babel macros (which myLib's author of course never intended).
Third, why is Jest trying to transpile that other library, which lies outside myApp, in the first place? It certainly shouldn't be. Actually, Jest tries to transpile everything besides what is explicitly excluded, and the exclusion under CRA defaults to (source):
transformIgnorePatterns: [
'[/\\\\]node_modules[/\\\\].+\\.(js|jsx|mjs|cjs|ts|tsx)$',
'^.+\\.module\\.(css|sass|scss)$',
],
The intent is to include the app itself and exclude its dependencies. This approach sort of assumes that all the dependencies of myApp will be somewhere under a node_modules, which (PnP aside) would be an especially reasonable expectation under nodeLinker: node-modules. But with a portal, there is a symbolic link to myLib, which is (after resolving its real path) not under any node_modules and therefore does not match the ignore patterns. Thus, dependencies accessed through portals are not excluded from Jest transformation under this config.
Solution
To solve it, configure Jest to also exclude anything not under the project's root directory, i.e., add:
'^(?!<rootDir>)'
Since I happen to be using Craco already, I can just:
// craco.config.js
module.exports = {
jest: {
configure: jestConfig => ({
...jestConfig,
transformIgnorePatterns: [
...jestConfig.transformIgnorePatterns,
'^(?!<rootDir>)',
],
}),
},
};
There really ought to be a built-in way to handle this, but 🤷♂️
I have just published a new TypeScript-based module to the NPM registry, ooafs. However, when I try to install it and import it in another TypeScript project, VSCode gives me the following error on the import statement: Cannot find module 'ooafs'.ts(2307).
This module's source files are compiled to JavaScript to a dist/ folder and definitions (.d.ts) are also generated.
Here's the tree of the published module (the one we download when we npm install):
.
├── dist
│ ├── Entry.d.ts
│ ├── EntryFilter.d.ts
│ ├── EntryFilter.js
│ ├── Entry.js
│ ├── EntryType.d.ts
│ ├── EntryType.js
│ ├── FSTypings.d.ts
│ ├── FSTypings.js
│ ├── index.d.ts
│ └── index.js
├── LICENSE
├── package.json
└── README.md
The package.json does contain the following entries:
{
"main": "dist/index.js",
"types": "dist/index.d.ts",
...
}
Because the module works normally on Runkit (pure JS), I assume the only problem I have is related to TypeScript, and it's not the first time TypeScript tells me a module doesn't exist when missing declaration files are the only problem.
Am I missing a step in the compilation process ?
Are my package.json properties wrong ?
If you need to see more code, the Github link is at the beginning of the question, and the published module structure can be found here: https://unpkg.com/ooafs#0.1.2/dist/.
Actually, the problem didn't come from my module (ooafs). It was a problem with the tsconfig.json of the project I was using the module in: The module property must be set to commonjs apparently.
Very late edit:
Also, I highly recommend setting esModuleInterop to true which allows you to import non-es6 modules in a more natural manner.
The answer is not the fix, and is certainly not ideal when you have to use top-level awaits (which don't work on commonjs).
You want to make sure your import path is the final file that node will try and load. So you cannot rely on folders resolving to folder/index.js and you cannot rely on giving file names without extensions (give the ".js" extension)
I want to create a library in Typescript that I can share via npm. Specifically, I want to use webpack to generate a js bundle along with a definition file to share the types with the js. So I'd have a tree of files like:
├── lib
│ ├── lib.d.ts
│ └── lib.min.js
├── test
...
├── ts
│ ├── errors
│ │ ├── CannotModifyAlteredObject.ts
│ ├── Lib.ts
│ ├── PostProcessors.ts
│ ├── Serializers.ts
├── tsconfig.json
├── typings.json
├── LICENSE
├── package.json
├── README.md
└── webpack.lib.config.js
And all the types exported by ts/Lib.ts would be exported to a single .d.ts in the lib directory to sit next to the js bundle.
I've looked at the following questions/sources:
Writing npm modules in typescript
How to create a typescript library (and the question it duplicates)
This unanswered question
The offical typescript guide to creating packages
This example typescript library project
And another SO question
However, none of these provide an example using webpack. Being able to bundle everything you need to use the library (apart from the nodejs runtime) into a single file is pretty important for my use case, so webpack fits this role well. I'd like to be able to generate a .d.ts file that maps to what webpack creates. However, I want to avoid creating the .d.ts file manually - it should be possible to automatically extract the types without having manually created .d.ts files get out of sync with my source code. Is there a way of doing this?
Typescript seems to be a better choice for writing code where the code base is huge and requires more consistency.
But the examples and experiences I find around the internet are more on the client side, may be coz of the Angular2.0 decision to use AtScript/Typescript.
Though Typescript adds a lot of power to Javascript, there is something that keeps on pissing me off. How do I manage codebase with duplicate files. My understanding is I have two ways to do it.
1st Way
Use a build tool like gulp/grunt, watch for changes, and compile Typescript to Javascript in the same folder. This may look something like this:
├── models/
│ ├── User.ts
│ ├── User.js
│ ├── Likes.ts
│ ├── Likes.js
2nd Way
Another way will be the same but of instead of outputting Javascript into the same folder, I can clone the entire workspace:
├── typescript
│ ├── models/
│ │ ├── User.ts
│ │ ├── Likes.ts
├── javascript
│ ├── models/
│ │ ├── User.js
│ │ ├── Likes.js
Both are not the best solution for me. Is there any other way to manage codebase for nodejs. For client it is simple as we have only one output file.
Is there any other way to manage codebase for nodejs. For client it is simple as we have only one output file
No. We personally try to go with the second.
With the JS excluded from the REPO and rebuilt pre npm publish.
I notice that Node.js projects often include folders like these:
/libs, /vendor, /support, /spec, /tests
What exactly do these mean? What's the different between them, and where should I include referenced code?
Concerning the folders you mentioned:
/libs is usually used for custom classes/functions/modules
/vendor or /support contains 3rd party libraries (added as git
sub-module when using git as source control)
/spec contains specifications for BDD tests.
/tests contains the unit-tests for an application (using a testing
framework, see
here)
NOTE: both /vendor and /support are deprecated since NPM introduced a clean package management. It's recommended to handle all 3rd-party dependencies using NPM and a package.json file
When building a rather large application, I recommend the following additional folders (especially if you are using some kind of MVC- / ORM-Framework like express or mongoose):
/models contains all your ORM models (called Schemas in mongoose)
/views contains your view-templates (using any templating language supported in express)
/public contains all static content (images, style-sheets, client-side JavaScript)
/assets/images contains image files
/assets/pdf contains static pdf files
/css contains style sheets (or compiled output by a css engine)
/js contains client side JavaScript
/controllers contain all your express routes, separated by module/area of your application (note: when using the bootstrapping functionality of express, this folder is called /routes)
I got used to organize my projects this way and i think it works out pretty well.
Update for CoffeeScript-based Express applications (using connect-assets):
/app contains your compiled JavaScript
/assets/ contains all client-side assets that require compilation
/assets/js contains your client-side CoffeeScript files
/assets/css contains all your LESS/Stylus style-sheets
/public/(js|css|img) contains your static files that are not handled by any compilers
/src contains all your server-side specific CoffeeScript files
/test contains all unit testing scripts (implemented using a testing-framework of your choice)
/views contains all your express views (be it jade, ejs or any other templating engine)
There is a discussion on GitHub because of a question similar to this one:
https://gist.github.com/1398757
You can use other projects for guidance, search in GitHub for:
ThreeNodes.js - in my opinion, seems to have a specific structure not suitable for every project;
lighter - an more simple structure, but lacks a bit of organization;
And finally, in a book (http://shop.oreilly.com/product/0636920025344.do) suggests this structure:
├── index.html
├── js/
│ ├── main.js
│ ├── models/
│ ├── views/
│ ├── collections/
│ ├── templates/
│ └── libs/
│ ├── backbone/
│ ├── underscore/
│ └── ...
├── css/
└── ...
More example from my project architecture you can see here:
├── Dockerfile
├── README.md
├── config
│ └── production.json
├── package.json
├── schema
│ ├── create-db.sh
│ ├── db.sql
├── scripts
│ └── deploy-production.sh
├── src
│ ├── app -> Containes API routes
│ ├── db -> DB Models (ORM)
│ └── server.js -> the Server initlializer.
└── test
Basically, the logical app separated to DB and APP folders inside the SRC dir.
Assuming we are talking about web applications and building APIs:
One approach is to categorize files by feature. To illustrate:
We are developing a library application. In the first version of the application, a user can:
Search for books and see metadata of books
Search for authors and see their books
In a second version, users can also:
Create an account and log in
Loan/borrow books
In a third version, users can also:
Save a list of books they want to read/mark favorites
First we have the following structure:
books
└─ entities
│ └─ book.js
│ └─ author.js
│
└─ services
│ └─ booksService.js
│ └─ authorsService.js
│
└─ repositories
│ └─ booksRepository.js
│ └─ authorsRepository.js
│
└─ controllers
│ └─ booksController.js
│ └─ authorsController.js
│
└─ tests
└─ ...
We then add on the user and loan features:
user
└─ controllers
└─ entities
└─ services
└─ ...
loan
└─ controllers
└─ ...
And then the favorites functionality:
favorites
└─ controllers
└─ entities
└─ ...
For any new developer that gets handed the task to add on that the books search should also return information if any book have been marked as favorite, it's really easy to see where in the code he/she should look.
Then when the product owner sweeps in and exclaims that the favorites feature should be removed completely, it's easy to remove it.
It's important to note that there's no consensus on what's the best approach and related frameworks in general do not enforce nor reward certain structures.
I find this to be a frustrating and huge overhead but equally important. It is sort of a downplayed version (but IMO more important) of the style guide issue. I like to point this out because the answer is the same: it doesn't matter what structure you use as long as it's well defined and coherent.
So I'd propose to look for a comprehensive guide that you like and make it clear that the project is based on this.
It's not easy, especially if you're new to this! Expect to spend hours researching. You'll find most guides recommending an MVC-like structure. While several years ago that might have been a solid choice, nowadays that's not necessarily the case. For example here's another approach.
This is indirect answer, on the folder structure itself, very related.
A few years ago I had same question, took a folder structure but had to do a lot directory moving later on, because the folder was meant for a different purpose than that I have read on internet, that is, what a particular folder does has different meanings for different people on some folders.
Now, having done multiple projects, in addition to explanation in all other answers, on the folder structure itself, I would strongly suggest to follow the structure of Node.js itself, which can be seen at: https://github.com/nodejs/node. It has great detail on all, say linters and others, what file and folder structure they have and where. Some folders have a README that explains what is in that folder.
Starting in above structure is good because some day a new requirement comes in and but you will have a scope to improve as it is already followed by Node.js itself which is maintained over many years now.
Just clone the repo from GitHub
https://github.com/abhinavkallungal/Express-Folder-Structure
This is a basic structure of a node.js express.js project
with already setup MongoDB as database, hbs as view engine, nodemon also,
so you can easily set up node js express project
Step 1: download or clone the repo
Step 2: Open in any code editor
Step 3: Open the terminal on the folder path
Step 4: run the comment in terminal npm start
Step 5: start coding