So I'm having a pretty odd issue. I have a python script whose function is to analyze an image and produce an output string. I decided to implement this script into the backend of my NodeJS server using the package 'python-shell.' The funny thing is, this script works perfectly when it's run on its own. It is able to analyze the image and produce the output string. However, when I attempt to have the 'python-shell' package run it, there is an error that is produced (related to the python code). I've tested and the python-shell runs the same version of python as is used at the terminal, so I'm not 100% sure why this issue is occurring. For reference, the below code is how I'm running the script:
var express = require('express');
var router = express.Router();
var pythonshell = require('python-shell');
pythonshell.defaultOptions = { scriptPath: '/path/to/myscript.py' };
var outputString = '';
var options = {args: ['-p', '/path/to/image.jpg']}
var pyshell = new pythonshell('myscript.py', options);
pyshell.on('message', function(message) {
// received a message sent from the Python script
console.log(message);
outputString = message;
});
pyshell.end(function(err) {
// This is where the error occurs
if (err) throw err;
console.log('Picture analysis complete');
console.log('outputString: ' + outputString);
});
Again, when the script is manually run (i.e. python myscript.py -p image.jpg), it runs perfectly fine. I can include the code in myscript.py, although I doubt that will help as it doesn't seem to be an error with the actual python code. The image is analyzed using several packages including OpenCV (and the produced error is, I believe, based off that). Any tips are greatly appreciated!
change the default path
pythonshell.defaultOptions = { scriptPath: '/path/to/myscript.py' };
Related
I created a test WASM program using Go. In the program's main, it adds an API to the "global" and waits on a channel to avoid from exiting. It is similar to the typical hello-world Go WASM that you can find anywhere in the internet.
My test WASM program works well in Browsers, however, I hope to run it and call the API using Node.js. If it is possible, I will create some automation tests based on it.
I tried many ways but I just couldn't get it work with Node.js. The problem is that, in Node.js, the API cannot be found in the "global". How can I run a GO WASM program (with an exported API) in Node.js?
(Let me know if you need more details)
Thanks!
More details:
--- On Go's side (pseudo code) ---
func main() {
fmt.Println("My Web Assembly")
js.Global().Set("myEcho", myEcho())
<-make(chan bool)
}
func myEcho() js.Func {
return js.FuncOf(func(this js.Value, apiArgs []js.Value) any {
for arg := range(apiArgs) {
fmt.Println(arg.String())
}
}
}
// build: GOOS=js GOARCH=wasm go build -o myecho.wasm path/to/the/package
--- On browser's side ---
<html>
<head>
<meta charset="utf-8"/>
</head>
<body>
<p><pre style="font-family:courier;" id="my-canvas"/></p>
<script src="wasm_exec.js"></script>
<script>
const go = new Go();
WebAssembly.instantiateStreaming(fetch("myecho.wasm"), go.importObject).then((result) => {
go.run(result.instance);
}).then(_ => {
// it also works without "window."
document.getElementById("my-canvas").innerHTML = window.myEcho("hello", "ahoj", "ciao");
})
})
</script>
</body>
</html>
--- On Node.js' side ---
globalThis.require = require;
globalThis.fs = require("fs");
globalThis.TextEncoder = require("util").TextEncoder;
globalThis.TextDecoder = require("util").TextDecoder;
globalThis.performance = {
now() {
const [sec, nsec] = process.hrtime();
return sec * 1000 + nsec / 1000000;
},
};
const crypto = require("crypto");
globalThis.crypto = {
getRandomValues(b) {
crypto.randomFillSync(b);
},
};
require("./wasm_exec");
const go = new Go();
go.argv = process.argv.slice(2);
go.env = Object.assign({ TMPDIR: require("os").tmpdir() }, process.env);
go.exit = process.exit;
WebAssembly.instantiate(fs.readFileSync(process.argv[2]), go.importObject).then((result) => {
go.run(result.instance);
}).then(_ => {
console.log(go.exports.myEcho("hello", "ahoj", "ciao"));
}).catch((err) => {
console.error(err);
process.exit(1);
});
This pseudo code represents 99% content of my real code (only removed business related details). The problem is that I not only need to run the wasm program (myecho.wasm) by Node.js, but I also need to call the "api" (myEcho), and I need to pass it parameters and receive the returned values, because I want to create automation tests for those "api"s. With Node.js, I can launch the test js scripts and validate the outputs all in the command line environment. The browser isn't a handy tool for this case.
Running the program by node wasm_exec.js myecho.wasm isn't enough for my case.
It would be nice to know more details about your environment and what are you actually trying to do. You can post the code itself, compilation commands, and versions for all the tools involved.
Trying to answer the question without these details:
Go WASM is very browser oriented, because the go compiler needs the glue js in wasm_exec.js to run. Nodejs shouldn't have a problem with that, and the following command should work:
node wasm_exec.js main.wasm
where wasm_exec.js is the glue code shipped with your go distribution, usually found at $(go env GOROOT)/misc/wasm/wasm_exec.js, and main.wasm is your compiled code. If this fails, you can post the output as well.
There is another way to compile go code to wasm that bypasses wasm_exec.js, and that way is by using the TinyGo compiler to output wasi-enabled code. You can try following their instructions to compile your code.
For example:
tinygo build -target=wasi -o main.wasm main.go
You can build for example a javascript file wasi.js:
"use strict";
const fs = require("fs");
const { WASI } = require("wasi");
const wasi = new WASI();
const importObject = { wasi_snapshot_preview1: wasi.wasiImport };
(async () => {
const wasm = await WebAssembly.compile(
fs.readFileSync("./main.wasm")
);
const instance = await WebAssembly.instantiate(wasm, importObject);
wasi.start(instance);
})();
Recent versions of node have experimental wasi support:
node --experimental-wasi-unstable-preview1 wasi.js
These are usually the things you would try with Go and WASM, but without further details, it is hard to tell what exactly is not working.
After some struggling, I noticed that the reason is simpler than I expected.
I couldn't get the exported API function in Node.js simply because the API has not been exported yet when I tried to call them!
When the wasm program is loaded and started, it runs in parallel with the caller program (the js running in Node).
WebAssembly.instantiate(...).then(...go.run(result.instance)...).then(/*HERE!*/)
The code at "HERE" is executed too early and the main() of the wasm program hasn't finished exporting the APIs yet.
When I changed the Node script to following, it worked:
WebAssembly.instantiate(fs.readFileSync(process.argv[2]), go.importObject).then((result) => {
go.run(result.instance);
}).then(_ => {
let retry = setInterval(function () {
if (typeof(go.exports.myEcho) != "function") {
return;
}
console.log(go.exports.myEcho("hello", "ahoj", "ciao"));
clearInterval(retry);
}, 500);
}).catch((err) => {
console.error(err);
process.exit(1);
});
(only includes the changed part)
I know it doesn't seem to be a perfect solution, but at least it proved my guess about the root cause to be true.
But... why it didn't happen in browser? sigh...
I use an NPM library to parse markdown to HTML like this:
var Markdown = require('markdown-to-html').Markdown;
var md = new Markdown();
...
md.render('./test', opts, function(err) {
md.pipe(process.stdout)
});
This outputs the result to my terminal as intended.
However, I need the result inside the execution of my node program. I thought about writing the output stream to file and then reading it in at a later time but I can't figure out a way to write the output to a file instead.
I tried to play around var file = fs.createWriteStream('./test.html'); but the node.js streams rather give me headaches than results.
I've also looked into the library's repo and Markdown inherits from Readable via util like this:
var util = require('util');
var Readable = require('stream').Readable;
util.inherits(Markdown, Readable);
Any resources or advice would be highly appreciated. (I would also take another library for parsing the markdown, but this gave me the best results so far)
Actually creating a writable file-stream and piping the markdown to this stream should work just fine. Try it with:
const writeStream = fs.createWriteStream('./output.html');
md.render('./test', opts, function(err) {
md.pipe(writeStream)
});
// in case of errors you should handle them
writeStream.on('error', function (err) {
console.log(err);
});
i have an error with redis set command on local redis server(127.0.0.1:6379)
versions:
npm version : 2.15.0;
node version : 4.4.2;
nodejs verison : 0.10.25;
redis version : 2.7.1;
Error:
events.js:141 throw er; // Unhandled 'error' event
ReplyError: ERR wrong number of arguments for 'set' command at parseError
(/opt/xxx/xxx/node_modules/redis/node_modules/redis
parser/lib/parser.js:193:12) at parseType
(/opt/xxx/xxx/node_modules/redis/node_modules/redis-
parser/lib/parser.js:303:14)
all of my codes look like this:
redis.set("key","value")
on my local machine the code is running successfully , but on aws linux machine i got this error.
var matchedMaps = map.get(publicURIField);
if(matchedMaps) {
matchedMaps.forEach(function(matchedMap){
var patternToValidate = matchedMap.pattern;
var type = matchedMap.type;
var tagID = matchedMap.tagID;
var patternToCheck = "cs-uri-stem";
var patternToSave = "";
if(type==1){
patternToCheck = "c-referrer";
}
var regexToFind = new RegExp(patternToValidate.substring(1,patternToValidate.length-1));
var matchedPattern;
if (regexToFind.test(rawLogParsed[patternToCheck].toString())) {
if (matchedMap.regexType=="&"){
matchedMap.patterns.forEach(function(patternObject){
var key = patternObject.pattern.split("=")[0];
var value = rawLogParsed[patternToCheck].toString().split(key)[1];
if(rawLogParsed[patternToCheck].toString().split(key)[1].split("&")){
value = rawLogParsed[patternToCheck].toString().split(key)[1].split("&")[0];
}
patternToSave += key+value+"&";
});
}else{
matchedMap.patterns.forEach(function(patternObject){
if(patternObject.pattern.indexOf("*")>-1){
patternObject.pattern = patternObject.pattern.replace(/\*!/g, '.*');
}
patternToSave += rawLogParsed[patternToCheck].toString().match(patternObject.pattern)+"/";
});
}
patternToSave = patternToSave.substring(0,patternToSave.length-1);
var matchedField = publicURIField,matchedPattern = patternToSave
,key = tagID + "_"+userID+"_"+ matchedField + "_" + matchedPattern + "_" + type + "_" + fixedMinuteNumber;
if (tagUsageInfo[startKeyForRedis+key] == undefined) {
var tagObject = {
pattern:matchedPattern,
matchedField:matchedField,
userID:userID,
tagName:matchedMap.tagName,
monthNumber:parseInt(mMonthToCheck),
minuteNumber: parseInt(fixedMinuteNumber),
hourNumber: parseInt(yearMonthDayHourToCheck),
dayNumber: parseInt(yearMonthDayToCheck),
tagID: tagID,
matchedPattern: matchedPattern,
totalRequests: 1,
totalEgress: parseInt(bytes),
totalTransfered: parseInt(bytes),
totalRest: parseInt(totalWorld),
totalUS: parseInt(totalUS)
}
if(isIngress){
tagObject.totalIngres += parseInt(bytes);
}
dbclient1.set(startKeyForRedis+"tagUsage_"+key,JSON.stringify(tagObject));
tagUsageInfo[startKeyForRedis+"tagUsage_"+key] = startKeyForRedis+key;
}
else {
dbclient1.get(startKeyForRedis+"tagUsage_"+key, function(err, tagObject) {
var tagObjectJson = JSON.parse(tagObject);
tagObjectJson.totalRequests += 1;
tagObjectJson.totalEgress += parseInt(bytes);
tagObjectJson.totalTransfered += parseInt(bytes);
tagObjectJson.totalRest += parseInt(totalWorld);
tagObjectJson.totalUS += parseInt(totalUS);
tagObjectJson.totalRequests += 1;
if(isIngress){
tagObject.totalIngres += parseInt(bytes);
}
dbclient1.del(startKeyForRedis+"tagUsage_"+key);
dbclient1.set(startKeyForRedis+"tagUsage_"+key, JSON.stringify(tagObjectJson));
});
}
}
});
}
any help?
1)If your trying to run redis on windows set accepts only two arguments cause the redis version issue
2)try latest version of redis on linux it will work
Try installing this version of Redis on windows from the following link. You can find more information here https://github.com/ServiceStack/redis-windows
This link provides three options to install Redis on windows
Option 1) Install Redis on Ubuntu on Windows
Option 2) Running the latest version of Redis with Vagrant
Option 3) Running Microsoft's native port of Redis
I personally prefer option 3.
Hope this helped. Thanks.
all of my codes look like this: [...]
It's not important how all of your code looks like. It's important how the specific line that caused the problem looks like but unfortunately you didn't include it.
The errors that you provided include some files and line numbers but you seem to have removed the ones that are related to your code. If you read those messages carefully then you should be able to know what lines those errors are related to and focus on those lines.
If the errors show up on a server and not on your desktop then I would suspect that maybe you're trying to use some environment variables or files on the file system to populate some variables in your program, and those are not available on the server resulting in putting undefined there.
You will surely find the problem when you add console.log() statements to every place where you want to access Redis so that you first print it and then call to Redis. That way at least you will know what data is causing the problem. I suspect that you are having some undefined values or something like that.
Remember that JSON.stringify(undefined) returns undefined instead of a valid JSON string. Something like that may be causing problems. Adding debug messages will help to narrow it down.
Some extra advice: You can use prefix parameter of the redis module then you will not have to add startKeyForRedis+ all over the place. You can set a prefix once and have it prepended automatically. See the docs:
https://www.npmjs.com/package/redis
I was doing learning to use KUE a nodejs library for job scheduling that uses redis for saving data.
I got this error while running client.js(that puts jobs in queue) and worker.js(that process the schedule jobs).
I was running worker before running the client and that is why this happened.
I reversed the order and everything went fine!
To fix this error on windows Redis from v.3 required.
That's why I've took zipped release 3.0.504 from here and now all is working.
Quite simple.
I have faced similar type of error which was because of older version of Redis. This is compatibility issue which fixes a bug in Redis after Redis 2.6.12. Make sure you install recent version of Redis v3.X.
I want to use an yeoman generator inside a NodeJS project
I installed yeoman-generatorand generator-git (the generator that I want use) as locally dependency, and, at this moment my code is like this:
var env = require('yeoman-generator')();
var path = require('path');
var gitGenerator = require('generator-git');
var workingDirectory = path.join(process.cwd(), 'install_here/');
generator = env.create(gitGenerator);
obviously the last line doesn't work and doesn't generate the scaffold.
The question: How to?
Importantly, I want to stay in local dependency level!
#simon-boudrias's solution does work, but after I changed the process.chdir(), this.templatePath() and this.destinationPath() returns same path.
I could have use this.sourcePath() to tweak the template path, but having to change this to each yeoman generator is not so useful. After digging to yo-cli, I found the following works without affecting the path.
var env = require('yeoman-environment').createEnv();
env.lookup(function() {
env.run('generator-name');
});
env.create() only instantiate a generator - it doesn't run it.
To run it, you could call generator.run(). But that's not ideal.
The best way IMO would be this way:
var path = require('path');
var env = require('yeoman-generator')();
var gitGenerator = require('generator-git');
// Optionnal: look every generator in your system. That'll allow composition if needed:
// env.lookup();
env.registerStub(gitGenerator, 'git:app');
env.run('git:app');
If necessary, make sure to process.chdir() in the right directory before launching your generator.
Relevant documentation on the Yeoman Environment class can be found here: http://yeoman.io/environment/Environment.html
Also see: http://yeoman.io/authoring/integrating-yeoman.html
The yeoman-test module is also very useful if you want to pass predefined answers to your prompts. This worked for me.
var yeomanTest = require('yeoman-test');
var answers = require('from/some/file.json');
var context = yeomanTest.run(path.resolve('path/to/generator'));
context.settings.tmpdir = false; // don't run in tempdir
context.withGenerators([
'paths/to/subgenerators',
'more/of/them'
])
.withOptions({ // execute with options
'skip-install': true,
'skip-sdk': true
})
.withPrompts(answers) // answer prompts
.on('end', function () {
// do some stuff here
});
I'm trying to create a node server that spawns phantomjs processes to create screenshots. The grab.js script works fine when executed and I've confirmed that it writes to stdout. Problem is the node code that spawns the process simply hangs. I've confirmed that phantomjs is in the path. Anyone know what might be happening here or how I might troubleshoot this?
Here's the phantomjs code (grab.js) that renders the page and writes the data to stdout:
var page = require('webpage').create(),
system = require('system'),
fs = require('fs');
var url = system.args[1] || 'google.com';
page.viewportSize = {
width: 1024,
height: 1200
};
page.open(url, function() {
var b64 = page.renderBase64('png');
fs.write('/dev/stdout', b64, 'w');
phantom.exit();
});
And here's the node code that spawns the phantom progress and prints the result (hangs):
var http = require('http'),
exec = require('child_process').exec,
fs = require('fs');
exec('phantomjs grab.js google.com', function(error, stdout, stderr) {
console.log(error, stdout, stderr);
});
I have had similar issues with exec and then switched to using spawn instead and it worked.
According to this article , Use spawn when you want the child process to return huge binary data to Node, use exec when you want the child process to return simple status messages.
hth
I had same problem, in my case it was not in nodejs, but in phantomjs (v2.1).
It's known problem when phantom`s open method hangs.
Also, found second link (I guess same person wrote) in which author points that requestAnimationFrame is not working well with tweenJs, which causes freezing. PhantomJS returns unixtimestamp but tweenjs expects it to be DOMHighResTimeStamp, and so on...
Trick is to inject request-animation-frame.js (which is also provided in that article)