TypeScript on AWS Lambda: to bundle imports (how?) or not to bundle? or: Runtime.ImportModuleError: Cannot find module '#aws-sdk/..." - node.js

I have the following lambda.ts code I'm trying to make running on an AWS Lambda:
import 'aws-sdk'
import { /* bunch of stuff... */ } from "#aws-sdk/client-cloudwatch-logs";
import {Context, APIGatewayProxyResult} from 'aws-lambda';
import {DateTime} from "luxon";
export const lambdaHandler = async (event: any, context: Context): Promise<APIGatewayProxyResult> => {
/* ... stuff ... */
}
which gets transpiled to:
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.lambdaHandler = void 0;
require("aws-sdk");
//import {CloudWatchClient} from "#aws-sdk/client-cloudwatch";
const client_cloudwatch_logs_1 = require("#aws-sdk/client-cloudwatch-logs");
const luxon_1 = require("luxon");
const lambdaHandler = async (event, context) => {
/* ... transpiled stuff ... */
}
When hitting the button, I'm getting this Response:
{
"errorType": "Runtime.ImportModuleError",
"errorMessage": "Error: Cannot find module '#aws-sdk/client-cloudwatch-logs'\nRequire stack:\n- /var/task/lambda.js\n- /var/runtime/index.mjs",
"trace": [
"Runtime.ImportModuleError: Error: Cannot find module '#aws-sdk/client-cloudwatch-logs'",
"Require stack:",
"- /var/task/lambda.js",
"- /var/runtime/index.mjs",
" at _loadUserApp (file:///var/runtime/index.mjs:951:17)",
" at async Object.UserFunction.js.module.exports.load (file:///var/runtime/index.mjs:976:21)",
" at async start (file:///var/runtime/index.mjs:1137:23)",
" at async file:///var/runtime/index.mjs:1143:1"
]
}
I played a lot with tsconfig.json, trying many things from Google / GitHub / SO / Rumors / Astrology / Numerology / Praying (monotheistic, pantheon-dwelling, neither..), but it only made me more confused.
For instance:
Using the tsconfig.json from Building Lambda functions with TypeScript still emits a single transpiled .js file without embedding the imported #aws-sdk/client-cloudwatch-logs module in the emitted output lambda.js file
Installing the AWS Common Runtime (CRT) Dependency
states that I need to npm install #aws-sdk/... (naturally), but doesn't explain anything beyond, which makes me think that maybe I shouldn't bundle them at all, but simply import them (in the assumption that they are pre-defined/loaded in AWS's Lambda's runtime)
Runtime is Node.js 16.x, Handler is lambda.lambdaHandler (emitted file is called lambda.js), and this is my current tsconfig.json:
{
"$schema": "https://json.schemastore.org/tsconfig",
"compilerOptions": {
"module": "commonjs",
"moduleResolution": "Node",
"target": "ES2022",
"sourceMap": true,
"lib": [
"ES2021"
],
"typeRoots": ["node_modules/#types"],
"outDir": "build",
"baseUrl": "src",
"strict": true,
"forceConsistentCasingInFileNames": true,
"esModuleInterop": true,
"resolveJsonModule": true,
"inlineSources": true,
"rootDir": "src",
"preserveConstEnums": true,
"isolatedModules": true,
"incremental": true,
"importHelpers": true
},
"exclude": [
"node_modules",
"**/*.test.ts"
]
}
So I'm trying to understand:
Do I even need to bundle those imported modules (such as #aws-sdk/client-cloudwatch-logs) at all, or they are already loaded by AWS Lambda's runtime?
If I do need to bundle, then how? do I need to use some bundler or is it just a matter of configuring tsconfig.json properly?
If bundler isn't mandatory, then how do I setup tsconfig.json to emit those 3rd-party modules?
If a bundler is mandatory, then can they all fit (WebPack, Babel, etc..)? or since no frontend (index.html) is involved, then not all of them can fit?

AWS SDK for JavaScript v3 (AKA modular) is not installed globally in the lambda execution context. You are using a v3 module (#aws-sdk/client-cloudwatch-logs) which is why it fails. AWS SDK v2 is installed globally, you are also using it (aws-sdk) so that require works fine.
You should use a bundler like webpack, esbuild, parcel or rollup. If you are using AWS CDK, there is a nodejs function construct that will do the bundling with esbuild for you.
TS will only emit your compiled javascript. If you are depending on javascript found in your node_modules directory, simply include that directory in your deployment package.
Generally, bundlers will take your application entry points (main.js, handler.js, whatever you want really) and recursively resolve all the dependencies, tree-shake any unreachable code then create one file for each entry point that has no other external dependencies. There is a runtime performance cost to this of course but it does simplify things and in a serverless context it isn't usually too impactful.
So, to resolve your error you can take one of two approaches:
Include your node_modules directory in your deployment package. (trivial)
Use a bundler or CDK (more complex)
Note that in either case, you need to be careful about dependencies with native bindings (binaries basically) as the one installed on your dev machine likely isn't supported in the lambda environment.

Related

Custom Dashboard for AdminJS not working in production

I have a Koa nodejs server which I added AdminJS to and it's working beautifully locally. My goal is to override the Dashboard component. I did so successfully when not running in production. However when I run in production mode (NODE_ENV=production node ./dist/server.js) it fails silently.
const componentLoader = new ComponentLoader();
const Components = {
Dashboard: componentLoader.add("Dashboard", "./admin/dashboard"),
};
const admin = new AdminJS({
componentLoader,
dashboard: {
component: Components.Dashboard,
}
});
My dashboard.tsx file is in src/admin/ and admin is a folder on the same level as src/server.ts. Also, my componentLoader when I inspect it is showing the correct filePath that ends with dist/admin/dashboard
Also, when I check dist/admin/dashboard.js I see my React code. So my tsconfig seems to be correct and the dashboard.tsx has a default export.
What confuses me is when I run nodemon --watch src --exec node -r esbuild-register src/server.ts is works correctly so it seems in general I have things hooked up correctly.
Lastly, here's my tsconfig.json.
{
"$schema": "https://json.schemastore.org/tsconfig",
"compilerOptions": {
"jsx": "react",
"lib": [
"es6"
],
"target": "es2017",
"module": "commonjs",
"esModuleInterop": true,
"resolveJsonModule": true,
"strict": true,
"allowSyntheticDefaultImports": true,
"noImplicitAny": true,
"allowJs": false,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"noImplicitReturns": true,
"strictNullChecks": true,
"moduleResolution": "node",
"inlineSources": true,
"sourceRoot": "/",
"sourceMap": true,
"isolatedModules": true,
"outDir": "./dist",
"rootDir": "./src",
"composite": true,
"baseUrl": ".",
"paths": {
"src/*": [
"src/*"
]
}
},
"exclude": [
"node_modules",
"./node_modules/*"
],
"files": [
"./src/server.ts"
],
"include": [
"./src/**/*",
"./src/*"
]
}
UPDATE:
I did notice that the components.bundle.js file was missing when navigating to my adminjs dashboard. Since I am using GCP App Engine, I know that that file will not able to be built and saved on the fly in the file system so I have integrated #adminjs/bundler which creates the missing files. However the piece I still haven't put together is how to integrate it into the build pipeline (in particular I'm not sure what the destination of the components.bundle.js should be).
Before I explain my solution here are a few pieces of context:
When using NODE_ENV=production, adminjs does a few things differently, in particular the components.bundle.js file gets served differently. In production, it looks for the file at ./.adminjs/bundle.js
That's when the bundler comes in (which is necessary anyway for certain cloud environments like GCP App Engine). You have to create your own components.bundler.js file which they have a tool for.
First, I created a file which bundles the frontend components. I have not tried doing that with the ComponentLoader so I wouldn't need duplicate code yet, but here's what I know for certain works:
import AdminJS, { OverridableComponent } from "adminjs";
const bundle = (path: string, componentName: string) =>
AdminJS.bundle(`./${path}`, componentName as OverridableComponent);
export const DashboardComponent = bundle("../src/dashboard", "Dashboard");
I believe if I were to create a file which creates the ComponentLoader and adds the components that it would be equivalent (it would export the Components and the componentLoader for use by the AdminJS configuration).
Note ../src/dashboard is simply the location of the dashboard.tsx file I chose. And Dashboard is the name of the component.
Then, I created a script which uses #adminjs/bundler to actually create the bundles. (I named it bundler.ts).
import { bundle } from "#adminjs/bundler";
/**
* yarn admin:bundle invokes this script.
* This file is used to bundle AdminJS files. It is used at compile time
* to generate the frontend component bundles that are used in AdminJS.
*/
void (async () => {
await bundle({
customComponentsInitializationFilePath: "./components.ts",
destinationDir: "./.adminjs",
});
})();
I added a script to my package.json which does the following:
ts-node ./bundler.ts && mv ./.adminjs/components.bundle.js ./.adminjs/bundle.js
Now, when I run this script (which I do when I run before doing node ./dist/server.js), the adminjs router is going to be able to find the previously missing file.
Note that when running your server you'll also want to make sure you set ADMIN_JS_SKIP_BUNDLE='true'.
I hope this helps the next person. I also do hope some documentation and better tooling is on its way. This is kind of messy but solved my issue for now.

Mongoose + Typescript - Unable to require file: mongodb\index.ts

I have a Node + Express application with Typescript, which was running fine, until I decided to include Mongoose in the equation. When I try running it, I get the following error:
TypeError: Unable to require file: mongodb\index.ts
This is usually the result of a faulty configuration or import. Make sure there is a '.js', '.json' or other executable extension with loader attached before 'ts-node' available.
I'm running the application with Nodemon, and have the following configuration in nodemon.json:
{
"execMap": {
"ts": "ts-node"
}
}
Here's my tsconfig.json:
{
"compilerOptions": {
"esModuleInterop": true,
"moduleResolution":"node",
"baseUrl": ".",
"target": "es6",
"paths": {
"#controllers/*": ["./controllers/*"],
"#services/*": ["./services/*"],
"#routes/*": ["./routes/*"],
"#customTypes/*": ["./types/*"],
"#utils/*": ["./utils/*"],
"#graphql/*": ["./graphql/*"]
}
}
}
I'm kind of new to Node with Typescript, so I probably made some mistakes, but cannot find any info regarding what exactly is wrong.
Tried downgrading Mongoose, installing MongoDB manually and changed versions of #types/mongoose, but to no avail.

How should I configure Typescript to compile code to run in Node.JS?

First, let me be clear that this is not related to the browser. There are a hundred different questions & answers about this that all start off with "you can't use CommonJS modules in the browser...", but that's not what I'm trying to do.
I have some Typescript code which I'm trying to compile to Javascript for execution via node file-scanner.js on a server. It seems like the configuration should be pretty straightforward, but I always wind up with an error: ReferenceError: exports is not defined.
I am using import/export exclusively in my code. The main entry point looks like this:
import { loadEnv } from './modules/environment';
import { getDatabase } from './modules/models/helpers/database';
import { indexFiles } from './modules/services/indexFiles';
loadEnv();
getDatabase()
.then((db) => {
return indexFiles(db);
})
.catch((err: Error) => {
console.error('[ERROR] Unable to index files');
console.error(`[ERROR] Message = ${err.message}`)
console.error('[ERROR] Error object:', err);
console.error('[ERROR] Stack Trace', err.stack || '--no trace available--');
});
And my tsconfig.json (copied from #tsconfig/bases node 10 configuration) looks like this:
{
"compilerOptions": {
"lib": ["es2018"],
"module": "CommonJS",
"target": "es2018",
"strict": true,
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true
},
"include": [
"src/file-scanner.ts",
"src/modules/**/*/ts"
],
"exclude": [
"src/**/*.spec.ts"
]
}
I'm building it with:
./node_modules/.bin/tsc -p tsconfig.file-scanner.json --outDir lib
And running it with:
node lib/file-scanner.js
From what I understand, the Typescript compiler should be converting all my import/export statements to CommonJS module.exports & require statements. And I would hope that it includes all the appropriate code. But for some reason, when it comes time to execute the code, exports is not defined.
There's probably something obvious here that's staring me in the face, but I can't seem to figure it out. How do I get Typescript to compile for execution via Node.JS?

ReferenceError: Node is not defined (trying to use Node interface in typescript function in nodejs application)

While extending Cheerio library, I implemented the following static function (other extension functions work fine):
$.nodeType = function (elem: CheerioElement): number {
switch (elem.type) {
case "comment":
return Node.COMMENT_NODE; // <--- it fails here
case "tag":
return Node.ELEMENT_NODE; // <--- it fails here
case "text":
return Node.TEXT_NODE; // <--- it fails here
default:
return -1;
}
};
The following error appears during runtime (compilation with tsc -b succeed):
ReferenceError: Node is not defined
Node interface is part of the DOM API. Thus, I realized the need of explicitly include the DOM API under compilerOptions section of tsconfig.json.
However, I still get that runtime error.
Minimal relevant part of tsconfig.json:
{
"compilerOptions": {
"baseUrl": ".",
"incremental": true,
"lib": [
"esnext",
"dom"
],
"module": "commonjs",
"noImplicitAny": true,
"outDir": "./lib/",
"sourceMap": true,
"target": "esnext",
"watch": true
},
"include": [
"./src/**/*.ts",
]
}
I thought of explicitly import Node lib in the specific .ts file which contains the function, but I didn't find any "include-able" standard DOM lib.
Including a typescript lib doesn't polyfill the feature in an environment where it is not available.
While including "dom" type definitions will make the types available (at the compile time), but it doesn't actually make the Node API (typically provided by the browser runtime) available at runtime.
If you really need this functionality at runtime, you will need to also include an implementation of DOM for node.js such as jsdom which provides this API.
lorefnon explained the problem; here's a slightly hacky way to fix it:
Install jsdom
npm install jsdom
Add this to the top of your file:
const { JSDOM } = require('jsdom'); // or import { JSDOM } from 'jsdom';
const Node = new JSDOM('').window.Node;

TypeScript AMD compilation and "barrel" modules

I'm trying to set up a Node.js + TypeScript project using Intern for testing. Everything works fine when I compile the project using "commonjs" (which I do for the normal build); and TypeScript is equally happy when compiling for "amd", which is required by Intern. However, when passing the tests with intern-client, it complains about a couple of things:
First, imports from "index.ts" files (so-called "barrel" modules) won't work. My setup is something like this (everything in the same directory):
// index.ts
export { x } from './x'
// x.ts
export function x() {}
// x.test.ts
import { x } from '.' // "Error: Failed to load module ..."
In fact, the generated JavaScript code (for x.test.ts) looks something like this:
define(["require", "exports", "."], function (...) { ... })
And I'm not sure that AMD knows how to handle the ".".
The second issue happens under the same circumstances (TypeScript compiles happily, but intern-client complains). In summary, I get an error when doing:
import jsdom = require('jsdom')
Which I need to transform to:
const jsdom = require('jsdom')
For Intern to be able to deal with it.
Here is the tsconfig.json file I use to compile the tests:
{
"compilerOptions": {
"target": "es6",
"module": "amd",
"moduleResolution": "node",
"sourceMap": true,
"rootDir": "src",
"outDir": "build/tests",
"noImplicitAny": true,
"suppressImplicitAnyIndexErrors": true
}
}
And here is my intern.js configuration file, in case it helps:
define({
suites: ['build/tests/**/*.test.js'],
excludeInstrumentation: true,
filterErrorStack: true
})
Edit (2017-05-03)
To help understand the issue, here is an excerpt of the directory tree of the project:
build
tests // The compiled tests will end up here
src
core
utils
x.ts
x.test.ts
// Other files, each containing a function that I would like to unit-test...
intern.js
package.json
tsconfig.json
...
Regarding the first issue, AMD's handling of an import like '.' is different than Node's. While both of them will map '.' to a package, Node uses a default module name of index.js, while AMD uses main.js. To get things working in an AMD loader, you'll need to first define a package for '.', and then tell the AMD loader what default module to use for that package. Given your project layout, you could configure Intern like this:
loaderOptions: {
map: {
// When a module in src/ references 'src/utils', redirect
// it to 'utils'
'src': {
'src/utils': 'utils'
}
},
packages: [
// Define a package 'utils' with files in 'src/utils' that defaults
// to the module index.js
{ name: 'utils', location: 'src/utils', main: 'index.js' }
]
}
Regarding the second issue, its not clear what the problem actually is. Import statements will be transpiled into define dependencies by TypeScript, so Intern should never be seeing them.

Resources