Exiftool can not be spawned in aws lambda environment, getting below error:
ERROR { Error: spawn /var/task/node_modules/dist-exiftool/node_modules/exiftool.pl/vendor/exiftool ENOENT
at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
at onErrorNT (internal/child_process.js:415:16)
at process._tickCallback (internal/process/next_tick.js:63:19)
errno: 'ENOENT',
code: 'ENOENT',
syscall:
'spawn /var/task/node_modules/dist-exiftool/node_modules/exiftool.pl/vendor/exiftool',
path:
'/var/task/node_modules/dist-exiftool/node_modules/exiftool.pl/vendor/exiftool',
spawnargs:
[ '-echo2', '1574158488325', '-stay_open', 'True', '-#', '-' ] }
A function is working properly with NodeJs version 8.10 on AWS Lamda but I want to upgrade this function to NodeJs 10.X version.
I failed to run a function on NodeJs 10.X. I am always getting an error on followings line.
const ep = new exiftool.ExiftoolProcess(exiftoolBin);
My function:
const exiftool = require('node-exiftool')
const exiftoolBin = require('dist-exiftool')
const fs = require('fs')
const path = require('path')
const ep = new exiftool.ExiftoolProcess(exiftoolBin)
const PHOTO_PATH = path.join(__dirname, 'photo.jpg')
const rs = fs.createReadStream(PHOTO_PATH)
ep.open()
.then(() => ep.readMetadata(rs, ['-File:all']))
.then((res) => {
console.log(res)
})
.then(() => ep.close(), () => ep.close())
.catch(console.error)
Please help me to resolve this issue.
Thank you.
The error message is rather misleading but there are some hints. The path says /exiftool.pl/, pl usually being perl. If you open the exiftool file you'll see #!/usr/bin/perl -w on the first line.
What's actually missing (ENOENT) is a perl binary
AWS apparently removed various binaries from the runtime environments. It seems the best practice now is using Layers to add missing dependencies.
So first, add perl to your AWS Lambda Function as a Layer, to do that you could use an ARN from https://metacpan.org/pod/AWS::Lambda. The perl binary should then be available at /opt/bin/perl, which doesn't match the #! of the script, so we have to adjust the function code:
const { ExiftoolProcess } = require('node-exiftool');
const exiftoolBin = require('dist-exiftool');
// Get perl binary path from environment variables or fallback to default.
const perlBin = process.env.PERL_BIN || '/opt/bin/perl';
// Instead of calling the perl script directly causing #! to choose the
// interpreter path, we'll pass the script path to the interpreter directly.
const exiftool = new ExiftoolProcess(`${perlBin} ${exiftoolBin}`);
But now the path passed to ExiftoolProcess is no longer a valid filepath. It's a shell command that is passed on to child_process.spawn(). Therefore we have to make sure it's handled as such by doing:
exiftool.open({ shell: true })
You might be done at this point, or you might run into the following error:
perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory. I'm not sure how to best resolve this problem. Maybe compiling perl yourself and creating a layer on your own prevents it. But the quick and dirty solution that worked for me was copying the libcrypt.so.1 file from my local ubuntu system ($ whereis libcrypt.so.1), putting it into a folder called lib, zipping that folder and uploading it as a new layer.
Related
When creating a CLI I would like to test my function. For that I'm using the module child_process.
const path = require('path');
const { execFile } = require('child_process');
describe('cli test', () => {
test('thing', () => {
const myCli = execFile(
`${path.resolve(__dirname, '..')}/cli.js`, ['--foo', 'Bar'],
(err, stdout, stderr) => {
if (err) {
console.log('err: ', err);
}
});
});
But this produces the following error:
Attempted to log "err: { Error: spawn /projects/cli/src/cli.js EACCES
at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
at onErrorNT (internal/child_process.js:415:16)
at process._tickCallback (internal/process/next_tick.js:63:19)
errno: 'EACCES',
code: 'EACCES',
Running this script directly in the terminal via the command: $ node cli.js --foo Bar works perfectly.
Now a suggestion is to chmod +x <file> that file (source). But the test should also work on CI, and on a different computer which pulls the Git repo.
Any idea?
I'd suggest using fork instead of execFile.
The child_process.fork() method is a special case of child_process.spawn() used specifically to spawn new Node.js processes.
This will allow you to execute JS files without needing them to be shell executable.
To the best of my knowledge, git actually tracks the executable bit for files. There are some things to consider though as pointed out in this article: https://medium.com/#tahteche/how-git-treats-changes-in-file-permissions-f71874ca239d
Another solution would be to not rely on the ./ execution syntax (which requires the executable bit to be turned on for the respective file) but instead to explicitly use the shell command:
const path = require('path');
const { execFile } = require('child_process');
describe('cli test', () => {
test('thing', () => {
const myCli = execFile(
`sh ${path.resolve(__dirname, '..')}/cli.js`, ['--foo', 'Bar'],
(err, stdout, stderr) => {
if (err) {
console.log('err: ', err);
}
});
});
Notice the sh prefix I added to your code, This way you thell the sh command (which should be available in all of your environments e.g. the CI) to execute the contents of the file, regardless of whether the file itself can be executed or not!
I was receiving an EACCESS -13 error from child_process.spawn when trying to run a the command line mysql command.
There was something wrong with my PATH and updating it to add /usr/local/mysql/bin/ resolved the problem.
The temporary fix is to run export PATH=$PATH:/usr/local/mysql/bin/.
The permanent fix is to:
type: sudo nano /etc/paths
Add /usr/local/mysql/bin at the end
Ctrl + X
Yes
Enter key
type hash -r # command line or close the terminal app and open it again
NOTE: I got the temporary fix from a site ... I don't know why it has a / on the end of the bin but all of the mysql executables appear to be available without it in the /etc/paths file
I am trying to write a file in node.js using fs.writeFile, I use the following code:
const fs = require('filer');
const jsonString = JSON.stringify(myObj)
fs.writeFile('/myFile.txt', jsonString, function (err) {
if (err) throw err;
console.log('Saved!');
});
}
I am sure the file is created, because I can read it by fs.readFile referring to the same address, but I cannot find it on the disk by using windows search. What I understood, if I change the localhost port it saves the files in another location. I already tried "process.cwd()", but it didn't work.
I really appreciate it if someone could help.
try to use : __dirname instead of process.cwd()
const fs = require('fs');
const path = require('path');
const filePath = path.join(__dirname, '/myFile.txt');
console.log(filePath);
const jsonString = JSON.stringify({ name: "kalo" })
fs.writeFile(filePath, jsonString, (err) => {
if (err) throw err;
console.log('The file has been saved!');
});
And I would like to know why are you using 'filer' instead of default fs module?
fs module is native module that provides file handling in node js. so you don't need to install it specifically. This code perfectly worked and it prints absolute location of the file as well.Just run this code if it doesn't work, I think you should re install node js. I have updated the answer.You can also use fs.writeFileSync method as well.
From documentation: "String form paths are interpreted as UTF-8 character sequences identifying the absolute or relative filename. Relative paths will be resolved relative to the current working directory as determined by calling process.cwd()."
So in order to determine your working directory (i.e. where fs create files by default) call (works for me):
console.log(process.cwd());
Then if you would like to change your working directory, you can call (works for me as well):
process.chdir('path_to_new_directory');
Path can be relative or absolute.
This is also from documentation: "The process.chdir() method changes the current working directory of the Node.js process or throws an exception if doing so fails (for instance, if the specified directory does not exist)."
Here is a simple example of adding command in nodejs using commander:
'use strict';
const {Command} = require('commander');
const run = () => {
const program = new Command();
console.log('CMD');
program.command('cmd [opts...]')
.action((opts) => {
console.log('OPTS');
});
program.parse(process.argv);
};
run();
In this case everything works fine, but when I'm adding description and options, commander throws an error:
program.command('cmd [opts...]', 'DESCRIPTION', {isDefault: true})
node test-commander.js cmd opts
test-commander-cmd(1) does not exist, try --help
My env:
node v8.9.3
npm 5.3.0
commander 2.12.2
That is the declared behavior of commander. From the npm page under Git-style sub-commands...
When .command() is invoked with a description argument, no .action(callback) should be called to handle sub-commands, otherwise there will be an error. This tells commander that you're going to use separate executables for sub-commands, much like git(1) and other popular tools.
The commander will try to search the executables in the directory of the entry script (like ./examples/pm) with the name program-command, like pm-install, pm-search.
So, when you add a description like you have, it'll assume you have another executable file called test-commander-cmd for the sub command.
If commander's behavior is not what you were expecting, I might recommend looking into a package I published, called wily-cli... only if you're not committed to commander, of course ;)
Assuming your code rests in file.js, your example with wily-cli would look like this...
const cli = require('wily-cli');
const run = () => {
cli
.command('cmd [opts...]', 'DESCRIPTION', (options, parameters) => { console.log(parameters.opts); })
.defaultCommand('cmd');
};
run();
// "node file.js option1 option2" will output "[ 'option1', 'option2' ]"
I'm trying to download files to the filesystem in an electron app. My code, in the main thread, looks like this:
const dir = `${__dirname}/media`;
if (!fs.existsSync(dir)){
fs.mkdirSync(dir);
}
const file = fs.createWriteStream(`${dir}/${name}`);
file.on("open", function() {
const request = http.get(url, function(response) {
response.pipe(file);
response.on('end', function() {
file.close();
...
});
});
request.on('error', function(err) {
...
});
});
This works when running in development using electron . But after I build it with electron-builder, I get the error in an alert:
Uncaught Exception:
Error: ENOENT, media/uploads_2016_02_BASF_Holistic_Program.jpg not found in /Users/nicholasstephan/Desktop/XXXXXXX/dist/Mac/XXXXXX.app/Contents/Resources/app.asar
at notFoundError (ELECTRON_ASAR.js:109:19)
at Object.module.(anonymous function) [as open] (ELECTRON_ASAR.js:209:16)
at WriteStream.open (fs.js:1890:6)
at new WriteStream (fs.js:1876:10)
at Object.fs.createWriteStream (fs.js:1831:10)
at next (/Users/nicholasstephan/Desktop/XXXXXXXX/dist/Mac/XXXXXXXX.app/Contents/Resources/app.asar/media.js:19:18)
at /Users/nicholasstephan/Desktop/XXXXXXXX/dist/Mac/XXXXXXXX.app/Contents/Resources/app.asar/media.js:52:4
...
where the media.js, ln 19, being referred to is the const file = fs.createWriteStream(${dir}/${name}); line in the code.
I've tried the solutions offered in about a dozen other similar stackoverflow answers, but none have fixed the problem.
What's going on here?
Thanks.
The built Electron app uses the Asar format. Asar is an archive format (it's really just one big file) though in Electron you are able to read from it as if it were a standard directory.
I presume (though I have not seen it explicitly documented) that it is not possible to write to an Asar with the fs functions. In any case there are almost certainly more appropriate locations to write data.
Try writing to a different path. Electron provides a number of useful paths using app.getPath(name) so you could for example write to the userData directory which holds configuration files for your app.
I'm on node v4.4.0 and on Windows 10. I'm using bunyan to log my node application.
try {
var fs = require('fs');
var path = require('path');
var spawn = require('child_process').spawn;
var through = require('through');
} catch (err) {
throw err;
}
var prettyStream = function () {
// get the binary directory of bunyan
var bin = path.resolve(path.dirname(require.resolve('bunyan')), '..', 'bin', 'bunyan');
console.log(bin); // this outputs C:\www\nodeapp\src\node_modules\bunyan\bin\bunyan, the file does exist
var stream = through(function write(data) {
this.queue(data);
}, function end() {
this.queue(null);
});
// check if bin var is not empty and that the directory exists
if (bin && fs.existsSync(bin)) {
var formatter = spawn(bin, ['-o', 'short'], {
stdio: [null, process.stdout, process.stderr]
});
// stream.pipe(formatter.stdin); // <- did this to debug
}
stream.pipe(process.stdout); // <- did this to debug
return stream;
}
The logging spits out in the console due to the fact I used stream.pipe(process.stdout);, i did this to debug the rest of the function.
I however receive the error:
Error: spawn C:\www\nodeapp\src\node_modules\bunyan\bin\bunyan ENOENT
at exports._errnoException (util.js:870:11)
at Process.ChildProcess._handle.onexit (internal/child_process.js:178:32)
at onErrorNT (internal/child_process.js:344:16)
at nextTickCallbackWith2Args (node.js:442:9)
at process._tickCallback (node.js:356:17)
at Function.Module.runMain (module.js:443:11)
at startup (node.js:139:18)
at node.js:968:3
I'm guessing this is a Windows error. Anyone have any ideas?
Use {shell: true} in the options of spawn
I was hit with this problem recently so decided to add my findings here. I finally found the simplest solution in the Node.js documentation. It explains that:
child_process.exec() runs with shell
child_process.execFile() runs without shell
child_process.spawn() runs without shell (by default)
This is actually why the exec and spawn behave differently. So to get all the shell commands and any executable files available in spawn, like in your regular shell, it's enough to run:
const { spawn } = require('child_process')
const myChildProc = spawn('my-command', ['my', 'args'], {shell: true})
or to have a universal statement for different operating systems you can use
const myChildProc = spawn('my-command', ['my', 'args'], {shell: process.platform == 'win32'})
Side notes:
It migh make sense to use such a universal statement even if one primairly uses a non-Windows system in order to achieve full interoperability
For full consistence of the Node.js child_process commands it would be helpful to have spawn (with shell) and spawnFile (without shell) to reflect exec and execFile and avoid this kind of confusions.
I got it. On Windows bunyan isn't recognized in the console as a program but as a command. So to invoke it the use of cmd was needed. I also had to install bunyan globally so that the console could access it.
if (!/^win/.test(process.platform)) { // linux
var sp = spawn('bunyan', ['-o', 'short'], {
stdio: [null, process.stdout, process.stderr]
});
} else { // windows
var sp = spawn('cmd', ['/s', '/c', 'bunyan', '-o', 'short'], {
stdio: [null, process.stdout, process.stderr]
});
}
I solved same problem using cross-spawn. It allows me to spawn command on both windows and mac os as one common command.
I think you'll find that it simply can't find 'bunyun', but if you appended '.exe' it would work. Without using the shell, it is looking for an exact filename match to run the file itself.
When you use the shell option, it goes through matching executable extensions and finds a match that way. So, you can save some overhead by just appended the executable extension of your binary.
I was having this same problem when trying to execute a program in the current working directory in Windows. I solved it by passing the options { shell: true, cwd: __dirname } in the spawn() call. Then everything worked, with every argument passed as an array (not attached to the program name being run).
I think, the path of bin or something could be wrong. ENOENT = [E]rror [NO] [ENT]ry