Unable to use any 3rd party module with AWS Lambdas - node.js

I am working on a lambda that makes use of modules (async, request, etc)
Unable to import module 'index': Error
at Function.Module._resolveFilename (module.js:338:15)
at Function.Module._load (module.js:280:25)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object.<anonymous> (/var/task/index.js:1:63)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
Sample code:
var
AWS = require('aws-sdk'),
util = require('util'),
request = require('request');
exports.handler = function(event, context) {
console.log('test');
context.done();
};
It works fine (prints test) as long as no 3rd party modules (besides aws-sdk) are required. As soon as I just add a line such as:
require('request') // or async, config and so on
It fails with the above error. I have tried calling these modules directly as well by specifying the full path with no luck. It's like its looking at the wrong directory when calling require.
Dumping process.env in the console yields:
PATH: '/usr/local/bin:/usr/bin:/bin',
LAMBDA_TASK_ROOT: '/var/task',
LAMBDA_RUNTIME_DIR: '/var/runtime',
AWS_REGION: 'us-west-2',
AWS_DEFAULT_REGION: 'us-west-2',
AWS_LAMBDA_LOG_GROUP_NAME: '/aws/lambda/Thumbnailer',
AWS_LAMBDA_LOG_STREAM_NAME: '2015/12/10/[$LATEST]3f8ef236195448c88f206634bde6301b',
AWS_LAMBDA_FUNCTION_NAME: 'Thumbnailer',
AWS_LAMBDA_FUNCTION_MEMORY_SIZE: '512',
AWS_LAMBDA_FUNCTION_VERSION: '$LATEST',
NODE_PATH: '/var/runtime:/var/task:/var/runtime/node_modules',
Here's the module I was working off - evidently this used to work at some point but does not for me.
Ideas? I feel that I am missing some configuration unique to lambdas here.

omg that was painful... Turns out that OSX makes the node_modules folder readable only by the user and AWS cannot read it. Make the node_modules folder and content readable by world and it works. I am not sure if all OSX setups react the same. I am using nvm which may be the culprit.
Update. I used to set all files to 0666 but ran into issues with executables. Here's a little script that will address things properly. It will set all files to 0666 unless executables or a directory in which case 0777. Run this from within the project folder (do be careful of the implications of not doing so!):
Here's a script from a question I posted:
#!/bin/bash
find . \
'(' -perm -0700 -exec chmod 0777 '{}' + ')' -o \
'(' -perm -0600 -exec chmod 0666 '{}' + ')'

Related

Expressjs change context where the server executes the app from

I'd like to know if it's possible to change the context from where the Express server is serving from, to avoid issues in the import/require files in the App that is served.
I'm serving from [root]/server.js, this file checks for the process.env.NODE_ENV and then requires a file in let's say ./dist/ENVIRONMENT/server.js
if (['staging', 'production'].indexOf(process.env.NODE_ENV) > -1) {
require('./dist/' + process.env.NODE_ENV + '/server.js')
} else {
require('babel-register')
require('./server.dev.js')
}
The files that are served are triggering an error related with the import/require locations that fails for the reason explained in the previous paragraph
Error: Cannot find module '../../../../xxxxx'
Why am I doing this? I'm transpiling the server source code for non development (that runs with babel) and then copied to the distribution directory. I understand that I can have tasks to rename and move this accordingly during deployment, that also solves the problem, but I'd like to learn or understand if we're able to change the context where the Express server, serves from!
From the top of my head, I guess I'll have to solve it through something like (which I'll be testing):
var dist = path.resolve(__dirname, '../../')
app.use(express.static(dist))
Which I attempted, but failed, same error:
Error: Cannot find module '../../../../config'
at Function.Module._resolveFilename (module.js:485:15)
at Function.Module._load (module.js:437:25)
at Module.require (module.js:513:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/Users/xxxx/www/projectFoobar/dist/staging/lib/services/foobarService/index.js:13:15)
at Module._compile (module.js:569:30)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:503:32)
at tryModuleLoad (module.js:466:12)
at Function.Module._load (module.js:458:3)
The config file happens to be in the [root] directory.
** OK just realised that using import/export is a bad practice
https://nodesource.com/blog/es-modules-and-node-js-hard-choices/

slimerjs Can not resolve required module encoding

I have seen slimerjs Can not resolve required module, works with phantomjs, but that one explains absolutely nothing, so I'll dare ask the question again.
I have done this:
$ npm install -g encoding
encoding#0.1.12 /home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/encoding
└── iconv-lite#0.4.13
So, apparently, it should be there?
Now, I'm trying this test script, as stated in https://www.npmjs.com/package/encoding - let's call it test_modload.js:
var encoding = require('encoding');
... and I try to run it with node:
$ node test_modload.js
module.js:338
throw err;
^
Error: Cannot find module 'encoding'
at Function.Module._resolveFilename (module.js:336:15)
at Function.Module._load (module.js:286:25)
at Module.require (module.js:365:17)
at require (module.js:384:17)
at Object.<anonymous> (/path/to/test/test_modload.js:1:78)
at Module._compile (module.js:434:26)
at Object.Module._extensions..js (module.js:452:10)
at Module.load (module.js:355:32)
at Function.Module._load (module.js:310:12)
at Function.Module.runMain (module.js:475:10)
Apparently, here I have to use NODE_PATH:
NODE_PATH=/home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules node test_modload.js
... and this command passes fine (i.e. returns nothing, and prints no errors).
But what if we try this with casperjs? The script becomes:
var encoding = require('encoding');
var casper = require('casper').create();
casper.run();
... and if I run with:
/home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs/bin/casperjs test_modload.js
... again all is fine (i.e. returns nothing, and prints no errors).
But let's try this with casperjs with slimerjs engine:
$ SLIMERJSLAUNCHER=/usr/bin/firefox46 /home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs/bin/casperjs --engine=slimerjs test_modload.js
Script Error: Module: Can not resolve "encoding" module required by main located at file:///home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs/bin/bootstrap.js
Stack:
-> file:///home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs/bin/bootstrap.js: 350
Eh... I guess NODE_PATH is missing?
$ NODE_PATH=/home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules SLIMERJSLAUNCHER=/usr/bin/firefox46 /home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs/bin/casperjs --engine=slimerjs test_modload.js
Script Error: Module: Can not resolve "encoding" module required by main located at file:///home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs/bin/bootstrap.js
Stack:
-> file:///home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/casperjs/bin/bootstrap.js: 350
Damn it, I ran out of options. What do I do now, how can I get encoding to work with slimerjs? Note that the slimerjs docs state in https://docs.slimerjs.org/current/api/require.html that one should use require.paths, however, for node standalone, it would complain with:
Error: require.paths is removed. Use node_modules folders, or the NODE_PATH environment variable instead.
... while slimerjs doesn't output this error; but still, no change if I do require.paths.push('/home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules'); - it still 'can not resolve "encoding" module'.
Just to make sure I have the right path, here is a listing:
$ ls /home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules
casperjs encoding npm utf8 zombie
$ ls /home/USERNAME/.nvm/versions/node/v4.0.0/lib/node_modules/encoding/
lib LICENSE node_modules package.json README.md test
... so apparently the module is there.
So, how on earth can I get slimerjs to find and use the encoding module in a script?

Parse-Server Local Installation — external modules

I installed with success a Parse-Server on my local machine from this GUID.
I tried first with global installation, then I tried to understand better (It's the first time I see something related to Node.js) and I installed in a local directory. I think it should be the same.
this is what I done:
npm install parse-server parse-dashboard underscore
this is how the directory looks like:
/parse: ls
dashboard-config.json
logs
node_modules
ls node_modules/underscore
LICENSE
README.md
package.json
underscore-min.js
underscore-min.map
underscore.js
ls node_modules/parse
parse/
parse-dashboard/
parse-json/
parse-server/
parseurl/
Next I try to include also the cloud code I developed. the main.js has this content:
//var Image = require("parse-image");
var _ = require('underscore');
...
This is how I started the server:
node_modules/parse-server/bin/parse-server \
--appId APPID --masterKey MASTERKEY \
--databaseURI mongodb://localhost:27017/MyAPP \
--cloud /absolutepathfor/MyApp/cloud/main.js
and the error I got
module.js:341
throw err;
^
Error: Cannot find module 'underscore'
at Function.Module._resolveFilename (module.js:339:15)
at Function.Module._load (module.js:290:25)
at Module.require (module.js:367:17)
at require (internal/module.js:16:19)
at Object.<anonymous> (/Users/......./main.js:2:9)
at Module._compile (module.js:413:34)
at Object.Module._extensions..js (module.js:422:10)
at Module.load (module.js:357:32)
at Function.Module._load (module.js:314:12)
at Module.require (module.js:367:17)
How can I include in this case underscore, but in any case other libraries?
[update]
I installed globally the underscore lib (node install -g underscore).
I create a symbolic link cloud -> < path where is located cloud/main.js >
Next I created package.json with npm init command
I launch again npm install
I created app.js like described in the guid and I configured it using the same parameters above.
I started the server with node app.js
all gone fine. the problem right now is in permissions on create a new document, where I should already have that grants, but this will be another problem to solve. I hope this can help somebody else
I hit the same problem and error message. I did the followings and it works
npm install underscore
use this path:
var _ = require('../node_modules/underscore/underscore.js')
Refer to more details in this post.
Can't get 'underscore' to work with parse server
I just sorted a similar issue while transferring Parse.com CloudCode to a self hosted Parse Server on Nodejs; in one of the controller we had the line
var _ = require('underscore.js');
It had to be done like this on Parse.com. But now as we are on NodeJS and have NPM—it's enough to just do:
var _ = require('underscore');
(ie: drop the .js extension)

TypeScript external node modules sometimes transpile to module.exports and exports

I'm in the process of converting my node app to use TypeScript external modules. Everything works fine when running the app but when converting some of my .ts files, the mocha tests "explode" due to SyntaxError: Unexpected reserved word.
After much debugging, I've found the following reproducible failure case. I have a simple autoRoles.ts file that defines the available user roles. Prior to using external modules, it looked like:
/// <reference path="../../typings/backend_typings.d.ts" />
module.exports.roles = {
// role definitions
}
Now after the conversion:
/// <reference path="../../typings/backend_typings.d.ts" />
export let roles = {
// role definitions
}
When running the mocha tests, it generates the following error:
>> Mocha exploded!
>> SyntaxError: Unexpected reserved word
>> at exports.runInThisContext (vm.js:53:16)
>> at Module._compile (module.js:413:25)
>> at Object.Module._extensions..js (module.js:452:10)
>> at Module.load (module.js:355:32)
>> at Function.Module._load (module.js:310:12)
>> at Module.require (module.js:365:17)
>> at require (module.js:384:17)
>> at Object.<anonymous> (/Users/abc/esupport/code/asp_v4/lib/models/userRole.ts:77:17)
>> at Module._compile (module.js:434:26)
>> at Object.Module._extensions..js (module.js:452:10)
>> at Module.load (module.js:355:32)
>> at Function.Module._load (module.js:310:12)
>> at Module.require (module.js:365:17)
>> at require (module.js:384:17)
>> at Object.<anonymous> (/Users/abc/esupport/code/asp_v4/lib/users/lib/ssoAuth.ts:7:17)
>> at Module._compile (module.js:434:26)
>> at Object.Module._extensions..js (module.js:452:10)
>> at Module.load (module.js:355:32)
>> at Function.Module._load (module.js:310:12)
>> at Module.require (module.js:365:17)
>> at require (module.js:384:17)
>> at Object.<anonymous> (/Users/abc/esupport/code/asp_v4/lib/users/index.ts:5:31)
I can flip between the old implementation and the new implementation for the autoRoles.ts file and get mocha to pass and fall, respectively. Note, there is a require('<path>/autoRoles') on line 77 of userRoles.ts.
When comparing between the transpiled versions, the only difference is that the old version uses 'module.exports' and the new one just has 'exports'.
Old:
/// <reference path="../../typings/backend_typings.d.ts" />
exports.roles = {
// role definitions
}
New:
/// <reference path="../../typings/backend_typings.d.ts" />
module.exports.roles = {
// role definitions
}
So I'm aware that "exports" is just a shortcut to "module.exports" so I can't explain why this is causing mocha to fail but I do know that if I switch between the two and change nothing else, mocha "explodes". I've also noticed that for other transpiled modules, tsc sometimes uses "module.exports" and sometimes uses "exports". Why the difference and more importantly, why is mocha exploding?
Unexpected reserved word
Have "use strict"; at the top of your file. You probably have a variable that is a reserved keyword. See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Strict_mode#Paving_the_way_for_future_ECMAScript_versions . TypeScript will warn on such variable names if you have that header in the file.
module.exports.roles = { is not the source of your error.
I've also noticed that for other transpiled modules, tsc sometimes uses "module.exports" and sometimes uses "exports".
Its similar to the nodejs convention. Basically saving characters (bytes) the runtime needs to parse.
export let foo = 123;
Will give you
exports.foo = 123;
(because exports == module.export therefore exports.foo == module.export.foo ... as you already know). However in :
let foo = 123;
export = foo;
It does
var foo = 123;
module.exports = foo;
Because if you reassign exports i.e. exports = foo then module.export !== exports. So you can use exports for extension .... but not assignment.
After more debugging, I found that mocha wasn't using the generated .js source code files that tsc transpiled. I'm not sure exactly how but it was trying to execute the "export var roles" located in the .ts file and "export" is a reserved word.
I came across this post, which indicated to me that mocha is trying to do it's own transpiling. That guy suggested to use "typescript-require" but that package looks like in the middle of being deprecated in favor of "ts-node". So I changed my grunt-ts config to look like:
mochaTest: {
test: {
options: {
reporter: 'spec',
require: [
'ts-node/register'
]
},
src: ['lib/test/**/*.spec.js']
}
},
This works but I'd be more than happy to have someone shed some light on what mocha was doing. Also, why was mocha successfully transpiling / not detecting "reserved word" in other .ts files that used export?
Edit 10/30/2015:
So I found out why mocha was trying to execute my .ts files. I was stupidly importing some of them as require('/path/file.ts') where I should have left the '.ts' extension off. I no longer need 'ts-node' in my mocha runner. This also explains why mocha was only erroring on some of my .ts files.

NODE_PATH is being ignored or not working

I'm trying to run my node app on a new server and am having some issues with the NODE_PATH environment variable. The app works fine on my local machine (OSX Lion) but not on the server (Redhat linux). When starting my app with node app.js from within my project directory /development/nodeproject, I get the following error :
Error: Cannot find module 'mod/core/models/Category'
at Function._resolveFilename (module.js:334:11)
at Function._load (module.js:279:25)
at Module.require (module.js:357:17)
at require (module.js:368:17)
at /development/nodeproject/app.js:57:5
at Object.<anonymous> (/development/nodeproject/app.js:258:1)
at Module._compile (module.js:432:26)
at Object..js (module.js:450:10)
at Module.load (module.js:351:31)
at Function._load (module.js:310:12)
mod/core/models/Category is the first require() in my app.js and looks like this: var Category = require('mod/core/models/Category'). So apparently node is not looking for modules in my project directory.
I'm not sure why though, because I made the following changes (that are working just fine on my local machine).
added export NODE_PATH=/development/nodeproject to my ~/.bash_profile
ran source ~/.bash_profile
if I run env I see NODE_PATH=/development/nodeproject listed there
in my app.js if I console log process.env.NODE_PATH I get /development/framework (should this output an array instead of a string?)
Other information that might be relevant:
I'm on node v0.6.7
I'm doing all of this as root (sudo su -)
At this point I'm not sure what else I can do. Any help would be greatly appreciated.
NODE_PATH used for modules, not for solutions files.
Try
module.paths.push("/development/nodeproject", "one/more/path");
before any require() call. And you really should use a relative require like require('./mod/core/models/Category') for files in your nodeproject directory
The functionality you are looking for was removed. Use the node_modules directory or a relative require like require('./mod/core/models/Category').
This answer has more info: NODE_PATH error with node.js while attempting to setup jsctags for vim

Resources