I am building a RESTful API with the serverless framework to run on AWS (API Gateway + Lambda + DynamoDB). It's my first Node project and my first serverless production project and I have no idea how to structure it. So far I have the following
|--Functions
|-----Function1
|--------InternalModule
|-----Function2
|-----Function3
|--------InternalModule
|-----Function4
|--Shared
|-----Module1
|-----Module2
|-----Module3
|--Tests
|-----Functions
|--------Function1
|-----------InternalModule
|--------Function2
|-----------InternalModule
|--------Function3
|-----------InternalModule
|--------Function4
|-----------InternalModule
|-----Modules
|--------Module1
|-----------InternalModule
|--------Module2
|-----------InternalModule
|--------Module3
|-----------InternalModule
I keep my API endpoints (Lambda handlers) in Functions. Some of them have internal modules which they only use and there are some who use modules from Shared. I want to have unit tests for all my modules - inner and shared as well as API testing on the lambda functions. I am using mocha and chai and want to integrate everything in a pipeline which on a git push runs the linters and tests and if they are successful deploy the API to the appropriate stage. The problem is that in order to test each module I have to have chai as a local node module in every folder where I have a test file and reference the modules to be tested by a relative path. In most cases it looks really ugly because of the nesting. If I want to test an internal module from
Tests/Functions/Function1/InternalModule
and I require it on top of the test like so
require('../../../../Tests/Functions/Function1/InternalModule')
+ I have to install chai in every folder so it's reachable. The same goes for mocha and all the dependencies needed for the test and I haven't even mentioned configuration. The main idea I am now considering is weather or not I should bring all modules to a folder called modules and require them when needed - worst case
from Functions/Function1
require('../../Modules/Module1')
Also keep the test files in the module folder and run them inside, but that would require the assertion library installed in every folder. I've read about npm link and symlinks but I want to keep track of what dependencies each folder has so I can install them on the CI environment after the clean project is downloaded from GitHub where I can't do links (or I've got the whole concept wrong?)
If anyone can suggest a better solution I would highly appreciate it!
the way Node uses require is so much more than I thought!
First, Node.js looks to see if the given module is a core module - Node.js comes with many modules compiled directly into the executable binary (ex. http, fs, sys, events, path, etc.). These core modules will always take precedence in the loading algorithm.
If the given module is not a core module, Node.js will then begin to search for a directory named, "node_modules". It will start in the current directory (relative to the currently-executing Javascript file in Node) and then work its way up the folder hierarchy, checking each level for a node_modules folder.
read more here
I will try out Putting all modules in a separate folder each with it's own Folder prefixed with FunctionName_ so I know where each module is used, and the test file + package.json file. Then if I need a module I can require it from the functions with shallow nesting which would look not so bad:
from Functions/Function1
require('module-1');
with package.json
"dependencies":{
"module-1":"file:../../Modules/Function1_Module1"
}
and have a separate folder Shared where I keep the shared Modules.
I am still open for better ideas!
Related
I am setting up CI/CD for a NodeJS project and occasionally the developer forgets to send up a file (module) to source control. I run npm ci and npm test without problem and the application gets deployed to my server. However, it will error out once executed due to the missing module.
Is there a best practice for ensuring that all files required by a node application are available before allowing it to be deployed?
I don't know your exact configuration but here is how my teams have handled similar issues in the past:
Unit Tests. Normally, a CI system can catch this type of error before you deploy. If your CI tests aren't flagging your missing module before the code gets deployed, then a commonly used solution would be to write a test that ensures the module is present. That way the problem would be automatically caught when your unit tests are run. Something along these lines might work:
// mymodule.test.js using Mocha syntax:
import {expect} from 'chai';
import mymodule from './mymodule';
describe('my module', () => {
it('should export something!', () => {
expect(!!mymodule).to.be.true;
});
});
Version Control workflow. It sounds like there is also an issue here with your team's version control workflow. Generally, all of the files required to build the production application should be kept under version control and developers should be committing frequently. In this situation, I would normally do some investigation to see what is happening -- it might require training or perhaps the app is structured in a way that is overly confusing or complex to engineers.
Use a lockfile for npm packages. If the missing module is a npm modules, then there are a few things that could cause it to be missing. Generally, all npm modules should be listed in your package-lock.json or yarn.lock file. This ensures that the production version of the application will be in sync with what developers are using locally. I personally discourage developers from installing npm modules globally unless it is absolutely necessary. In that situation, your CI server (and perhaps your production server) will need to be updated to include the exact same versions of the global packages.
Automated build systems. I think you are indicating that your problem is caused by modules that aren't under source control. But I have also seen some situations where legacy build systems might omit a file that is important when building the app for production. Modern build tools like Webpack and Babel normally include every module referenced by the application but older solutions like grunt and gulp might require some fine-tuning to ensure the files are always included in an automated way (to avoid a situation where developers are expected to manually add the modules to the build system and often forget to do so).
The best way to prevent this is to have the developer get the project's checksum and compare that with source control and/or your server. If the checksums match, then all of the files were transfered.
If you (or your deployment software/service) are using rsync in your deployment process and use the -C parameter then some directories might be getting filtered out.
I had a similar CI/CD issue where npm packages used the directory name "core" and it was ignored due to the -C parameter.
Replace all of your require() calls with webpack import calls, and have your build run webpack. At runtime, node will run the bundle instead of your regular entrypoint.
Webpack will catch all unreachable imports during build time.
All this is assuming the missing files are modules (code) rather than resources (e.g. JSON files).
I am working on a project that is hosted as a monorepo. For simplification purposes let's say that inside there are three self-explanatory packages: server, a webapp client and library. The directory structure would be something like the following:
the-project
packages
server
src
webapp
src
library
src
All packages employ flow type notation, use a few >ES5 features and, for this reason, go through babel transpilation. The key difference is that transpilation of the webapp package is done via webpack, whereas server employs a gulp task that triggers script transpilation through the gulp-babel package. library is transpiled automatically when web is built.
Now, the problem I have is that for server to build, babel requires library to be built first and its package.json to specify its (built) main JS source file so its transpiled artifacts can be included. As you can imagine, this would quickly become problematic if the project were to contain multiple libraries that are actively being developed (which it does), as all would require building, including any dependent packages (like server in this simple case).
As an attempt to overcome this annoyance, I initially thought of using webpack to build the server, which would take care of including whatever dependencies it requires into a bundle, but I ran into issues as apparently webpack is not meant to be used on node JS applications.
What strategies are available for building a node JS application requiring Babel transpilation, such that the application's source files as well as any dependencies are built transparently and contained in a single output directory?
Annex A
Simplified gulp task for transpilation of scripts, as employed by server.
return gulp
.src([`src/**/*.js`], { allowEmpty: true })
.pipe(babel({ sourceMap: true }))
.pipe(gulp.dest('dist'));
As can be seen above, only server's own source files are included in the task. If src were to be changed to also include library, the task would emit the dependencies' artifacts in server's own output directory and any require('library') statements within would attempt to locate the built artifacts in packages/library and not packages/server/dist, thus resulting in import failures.
First of all, I am not sure what your server is doing. If it is doing a database connection or some calculations then I would not recommend it to be built by webpack. Whereas If your server is just doing Server-Side Rendering and making some API calls to other servers then I would recommend it to be bundled using webpack.
A lot of projects follow this philosophy. For example, you can take a look at something similar, I have done in one of my personal projects [Blubus]. Specifically, you might be interested in webpack-server-config. And also you can take a look at how big projects like spectrum does it.
so far i've learned a bit about NodeJS. But now i want to write a huge enterprise app with it and i'm wondering how to setup the structure correctly? Coming from other languages like PHP and Java, i imagine, i would split my project in different NPM modules. For example #mybigproject/customer, #mybigproject/cart and #mybigproject/checkout and so on.
But those submodules would be installed in the node_modules folder of the application skeleton. How would i tell for example Express, that the template files are in the different module directories? Or for example i use TypeORM for data access. So each module would have it's own set of models. How do those models know the database configuration data, as it's only in the main application skeleton, or the other way around, how does the application skeleton should know where to find the models?
Don't use npm modules for different parts of your project.
This components is integral part of your project and usually depend on your global config / schema / routing / etc
Just put it in different files and require it where you need it.
You can get an idea for folders structure from projects like Sail.JS
Use npm modules if you writing some utility that going to serve you for different apps and you want an easy way to upgrade the utility code once for all your apps (or in case you want to share that utility as open source for all of us)
NPM can install your local folder as a dependency. (ref)
npm install <folder>:
Install the package in the directory as a symlink in the current
project. Its dependencies will be installed before it's linked. If
sits inside the root of your project, its dependencies may be
hoisted to the toplevel node_modules as they would for other types of
dependencies.
Your module keeps its original location after installed and a symlink is created as the same name of your module folder in the top level node_modules folder.
In these custom sub-modules, you can use __dirname and relative paths to locate you configuration files to feed to database or other data consumers.
But remember that, sub-modules often serve as utility functions for the main module. They should be independent from the project context.
I'm currently working on a React application that's consuming a React library we also develop.
Currently, the process is to copy over the dist folder of the library over to the node_modules folder of the application.
To resolve the tedious nature of this, I thought the solution would be simple: to npm link the package in our application, and have the JSX/React components run through the application's babel-loader. That way, we'd also get webpack's dev server to watch for changes in the library and refresh automatically.
The problem with this is that the library's babel settings are different from those of the consuming application. For instance, root imports in the library (e.g. import ~/some-module) are supposed to resolve from the root folder of the library, but instead, they resolve to the root folder of the application, resulting in errors, because the only babel configuration it uses is the .babelrc from the application.
I tried adding separate webpack config rules to make exceptions for the library, but now it feels kind of hacky. In addition to that, the webpack dev server runs incredibly slow to boot up, presumably because it's running a babel transformation on the library too.
Is there an easier way to do this? Like telling webpack that "for this library in node_modules, use its own configuration file, and respect all of its own babel settings and relative imports?"
I have a repo which consists of several "micro-services" which I upload to AWS's Lambda. In addition I have a few shared libraries that I'd like to package up when sending to AWS.
Therefore my directory structure looks like:
/micro-service-1
/dist
package.json
index.js
/micro-service-2
/dist
package.json
index.js
/shared-component-1
/dist
package.json
component-name-1.js
/shared-component-2
/dist
package.json
component-name-2.js
The basic deployment leverages the handy node-lambda npm module but when I reference a local shared component with a statement like:
var sharedService = require('../../shared-component-1/dist/index');
This works just fine with the node-lambda run command but node-lambda deploy drops this local dependency. Probably makes sense because I'm going below the "root" directory in my dependency so I thought maybe I'd leverage gulp to make this work but I'm pretty darn new to it so I may be doing something dumb. My strategy was to:
Have gulp deploy depend on a local-deps task
the local-deps task would:
npm build --production to a directory
then pipe this directory over to the micro-service under the /local directory
clean up the install in the shared
I would then refer to all shared components like so:
var sharedService = require('local/component-name-1');
Hopefully this makes what I'm trying to achieve. Does this strategy make sense? Is there a simpler way I should be considering? Does anyone have any examples of anything like this in "gulp speak"?
I have an answer to this! :D
TL;DR - Use npm link to link create a symbolic link between your common component and the dependent component.
So, I have a a project with only two modules:
- main-module
- referenced-module
Each of these is a node module. If I cd into referenced-module and run npm link, then cd into main-module and npm link referenced-module, npm will 'install' my referenced-module into my main-module and store it in my node_modules folder. NOTE: When running the second npm link, the name of the project is the one you find in your package.json, not the name of the directory (see npm link documentation, previously linked).
Now, in my main-module all I need to do is var test = require('referenced-module') and I can use that to my hearts content. Be sure to module.exports your code from your referenced-module!
Now, when you zip up main-module to deploy it to AWS Lambda, the links are resolved and the real modules are put in their place! I've tested this and it works, though not with node-lambda yet, though I don't see why this should be a problem (unless it does something different with the package restores).
What's nice about this approach as well is that any changes I make to my referenced-module are automatically picked up by my main-module during development, so I don't have to run any gulp tasks or anything to sync them.
I find this is quite a nice, clean solution and I was able to get it working within a few minutes. If anything I've described above doesn't make any sense (as I've only just discovered this solution myself!), please leave a comment and I'll try and clarify for you.
UPDATE FEB 2016
Depending on your requirements and how large your application is, there may be an interesting alternative that solves this problem even more elegantly than using symlinking. Take a look at Serverless. It's quite a neat way of structuring serverless applications and includes useful features like being able to assign API Gateway endpoints that trigger the Lambda function you are writing. It even allows you to script CloudFormation configurations, so if you have other resources to deploy then you could do so here. Need a 'beta' or 'prod' stage? This can do it for you too. I've been using it for just over a week and while there is a bit of setup to do and things aren't always as clear as you'd like, it is quite flexible and the support community is good!
While using serverless we faced a similar issue, when having the need to share code between AWS Lambdas. Initially we used to duplication the code, across each microservice, but later as always it became difficult to manage.
Since the development done in Windows Environment, using symbolic links was not an option for us.
Then we came up with a solution to use a shared folder to keep the local dependencies and use a custom written gulp task to copy these dependencies across each of the microservice endpoints so that the dependency can be required similar to npm package.
One of the decisions we made is not to keep two places to define the dependencies for microservices, so we used the same package.json to define the local shared dependencies, where gulp task passes this file and copy the shared dependencies accordingly also installing the npm dependencies with a single command.
Later we made the code open source as npm modules serverless-dependency-install and gulp-dependency-install.