AWS Lambda layer and local structure - node.js

I'm facing a problem on how configuring my local environment to reproduce the behaviour of the prod env.
A simplified view of my prod env :
├── λf A
│ └── layer L
│
├── λf B
│ └── layer L
│
└── λf C
A,B & C are classic node.js lambda functions. A & B share some common dependencies (let's say lodash.js) that I want to group under a lambda layer : L.
For my dev env, I'm using Lerna (but that's not mandatory) to work as a mono repo with this structure :
.
├── packages
│ ├── A
│ │ ├── node_modules
│ │ └── package.json
│ ├── B
│ │ ├── node_modules
│ │ └── package.json
│ ├── C
│ │ ├── node_modules
│ │ └── package.json
│ └── L
│ ├── node_modules
│ └── package.json
├── package.json
└── lerna.json
To ship L, all I have to do is installing it's dependencies and copying the content of node_modules folder into nodejs/node_modules path of my layer.
But what I'm currently unable to do it's making the module A & B to resolve their dependencies both in standard node_modules paths and the L node_modules folder.
Some solutions that are not acceptable :
Duplicating the dependencies
Adding the L dependencies globally in the lerna package.json (because in my real environment I have multiple layers)

Technically, that would be the complex case when you are making environment based on Lambda.
It has separate running environment, so it is difficult to regard same as traditional environment.
But there is easier path that AWS provides.
How about creating your lambda function from Container Image?
Basically, Lambda Layer is "Hidden" file system layer in Lambda Virtual Machine. So it would work very similar to containerized layered package.
If you build your serverless function with container package, you can build your development environment same as production environment.
Reference :
https://docs.aws.amazon.com/lambda/latest/dg/images-create.html
You can follow this out of the box example to start :
https://github.com/jinspark-lab/lambdakit

Related

Gitlab Releases with specific files or folders instead of the whole repository

Is there an option to pick specific files to be only included in a gitlab release instead of having to have the whole repository packaged as zip/tar files.
.
├── docker-compose.yml
├── Dockerfile
├── flow-definitions/ -- folder of flow definition
│ ├── main_flow_definition.json -- root flow definition for nifi instance
│ ├── bin/ -- folder for flow definitions impacted by dev cycle
│ │ ├── impacted_flow_definition.json
│ │ ├── impacted_flow_definition_2.json
│ │ └── ...
│ └── scripts/ -- scripts relevant to data pipelines
│ ├── databases/ -- DDL for target databases used in ETL
│ │ ├── db_name/
│ │ │ ├── <DDL scripts>
│ │ │ └── ...
│ │ └── ...
│ └── groovy/ -- Groovy scripts used in NiFi processors
│ ├── df_folder/
│ │ ├── <groovy scripts used in processors>
│ │ └── ...
│ └── ...
├── README.md
└── .gitignore
Currently I do the releases in gitlab through the UI and it packages the whole repository when a release is done. As you can see from the above folder structure majority of the time the changes are only within the bin/ folder. As the docker file and other components don't usually change frequently.
So instead of packaging all the files, it would be good to release versions that just have the specific changed files in this case within the bin/ folder.
I couldn't find a way to actually achieve this through gitlab if that's an option available

App Engine standard build on Cloud Build with npm#7

I have monorepo setup and upgraded from yarn 2 to npm 7.
Was working perfectly fine till I tried to deploy my application to the standard app engine environment.
gcloud app deploy kicks off a build inside Cloud Build, which builds my project via buildpacks (nodejs14 buildpack).
My problem is, that this buildpack includes nodejs14 and npm#6 and without using the flex environment it doesn't seem that there is a way to use npm#7 in this standard app engine cloud build.
What I've tried by now:
Use same OS and install node_modules, tar them, upload them, untar them. (could not really do it)
Link packages locally before uploading (don't know if I did that right. App was not finding local packages afterwards)
Trying to use a cloudbuild.yaml, but it seems that this is not possible with standard environment.
in the custom build step "gcp-build": "npm i -g npm#7" (does not work, permission error, see full build log here)
I would really appreciate every pointer in any direction that could help me out.
This is my project setup:
[project]/
├── packages/
│ ├── app
│ │ ├── node_modules/
│ │ ├── src/
│ │ ├── app.yaml
│ │ ├── ...
│ │ └── package.json
│ ├── shared
│ │ ├── node_modules/
│ │ ├── src/
│ │ ├── ...
│ │ └── package.json
│ ├── shared-admin
│ │ ├── node_modules/
│ │ ├── src/
│ │ ├── ...
│ │ └── package.json
│ ├── base-server
│ │ ├── node_modules/
│ │ ├── src/
│ │ ├── ...
│ │ └── package.json
├── package.json
│
└── node_modules/

Bazel using with lerna and yarn workspace

Many people are using lerna and/or yarn workspace.
I guess either migrating from them to Bazel, or just using them with Bazel together is good to be guided with an example project.
For example, currently, I have a directory structure like this, where foo is an express server and bar is a library consumed by foo, both based on typescript.
<project root>
├── jest.config.js
├── lerna.json
├── package.json
├── packages
│ ├── bar
│ │ ├── jest.config.js
│ │ ├── package.json
│ │ ├── src
│ │ │ └── index.ts
│ │ ├── test
│ │ │ └── unit
│ │ │ └── index.test.ts
│ │ ├── tsconfig.build.json
│ │ └── tsconfig.json
│ └── foo
│ ├── jest.config.js
│ ├── package.json
│ ├── src
│ │ ├── hello.ts
│ │ └── index.ts
│ ├── test
│ │ ├── integration
│ │ │ └── index.test.ts
│ │ └── unit
│ │ └── index.test.ts
│ ├── tsconfig.build.json
│ └── tsconfig.json
├── tsconfig.build.json
├── tsconfig.json
└── yarn.lock
How should I align it with Bazel, like you know, WORKSPACE, BUILD, and their contents?
Any tips or examples?
Thanks!
There are some examples of repo structures somewhat similar to this in the rules_nodejs examples directory. This shows (in this case an Angular app) having shared libs and consuming them, but the principle is the same here.
Generally, you'd only have one WORKSPACE file in the root of your project. While it's possible to have multiple package.json files for different apps and libs, it adds some extra complexity to the ts_library rules which, for getting started, may be best avoided. This example repo shows multiple package.json files, but without Typescript.
For BUILD (or BUILD.bazel) files, the minimum you'll need here is one in foo and one in bar (and one at the root). The more BUILD files you have, the more you split up the compilation units for your source, therefore increasing incrementality.
Then add ts_library rules to those BUILD files, docs for which can be found here, they also show the differences between using tsc directly and ts_library. You can then define source dependencies between foo and bar, a quick example shown below:
packages/foo/BUILD:
ts_libaray(
name = "foo",
srcs = glob(["src/**/*.ts"]),
deps = [
"//packages/bar", <-- this is the source dep for bar
"#npm//some-package",
],
)
packages/bar/BUILD:
ts_libaray(
name = "bar",
srcs = glob(["src/**/*.ts"]),
deps = [
"#npm//some-other-package",
],
)

How to use global/common node_modules instead of local node_modules?

I want to use the global node_modules (C:\Users\<user_name>\AppData\Roaming\npm\node_modules) or common node_modules (D:\node_modules) instead of local node_modules (D:\angular\projects\project1\node_modules).
I want to use the node_modules for various projects.
This is my project structure.
D:\
├── node_modules (common for all projects)
│ ├── Module-N
├── projects
│ ├── project1
│ │ ├── src
│ │ ├── app
│ │ | ├── Module-N (importing from common)
|
│ ├── project2
│ │ ├── src
│ │ ├── app
│ │ | ├── Module-N (importing from common)
|
├── ....
try working with relative path like when you require you own modules for the global models
for example
assuming we want to get a module from the global node modules and the file we are working on is seats on projects/project1/src
we will do require('../../../node_modules/my_global_model')
try looking into tsconfigs paths: https://www.typescriptlang.org/docs/handbook/module-resolution.html

How to avoid nesting of "node_module" directory?

In some cases, that comes into conflict with the Windows 260 chr path limitation. I'm having a lot of problems with npm install and this limitation.
.
├── app
│ └── node_modules
│ └── submodule
│ └── node_modules
│ └── submodule
│ └── node_modules
│ └── submodule
│ └── node_modules
│ └── submodule
│ └── to_infinity_and_beyond...
│ └── It's a madness!
...
That structure produces paths like:
c:/path_to_my_app/
node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/node_modules/sub_module/.....
What i'm doing wrong? It's there a way to avoid it?
Perhaps renaming "node_module" to "nm", or something like that, may helps to save some characters...
That's in advance!
As Brandon Tilley said:
npm dedupe
works fine for me!

Resources