In an electron application, I spawn a shell process proc to which the application
then talks via pipes, i.e. proc.stdin.write("cmd_to_shell_proc") and
proc.stdout.on("data", (data) => {//do sth with response}).
The data contains particular tokens from which it is clear what command had
been sent to the process. For most responses it is enough to know to which
command they reply, but there is one command I send from
two different components in my UI for which I would like to handle the
responses differently, depending on the UI component that sent it. I currently process the responses on the basis
of the string token contained in the data sent back from the process,
and I have not found a way to propagate any information about the originally calling process into the data handling in proc.stdout.on("data", ...) .
Is there any way to attribute the data being returned from the shell process to
the original caller of proc.stdin.write("<some command>")?
Any suggestions or advice would be highly appreciated!
Related
I am new to node-red and I am confused with the "message payload flow system".
I want to send a POST request that contains, among other params, files into the request payload. These files should be in an array called "files".
I read my files from my file system, this works fine, but in the function node, how do I build my POST payload?
So far I have this:
The problem is that the payload contains both files and I can't find a way to get them separately. How can I retrieve both my files, separately, into the BUILD-POST-REQ function?
The core Join node can be used to combine the output of parallel input streams. It has a number of modes that control how many input messages it will collect together.
These include a count of messages.
You can also choose how it combines the input messages, this can be as an array or as an object using the msg.topic as the key to the incoming msg.payload
Ok. I found A solution. But I don't know if this is a best practice. Feel free to correct me!
The idea is that after each file read, I store it in a new property of the msg object and then can access it later in the flow.
NodeJS is asynchronous, so for example if running an Express server, one might be in the middle of servicing one request, log it, and then start servicing another request and try to log it before the first has finished.
Since these are log files, it's not a simple write. Even if a write was atomic, maybe another process actually winds up writing at the offset the original process is about to and it winds up overwriting.
There is a synchronous append function (fs.appendFile) but this would require us to delay servicing a request to wait for a log file write to complete and I'm still not sure if that guarantees an atomic append. What is the best practice for writing to log files in NodeJS while ensuring atomicitiy?
one might be in the middle of servicing one request, log it, and then start servicing another request and try to log it before the first has finished.
The individual write calls will be atomic, so as long as you make a single log write call per request, you won't have any corruption of log messages. It is normal, however if you log multiple messages while processing a request, for those to be interleaved between many different concurrent requests. Each message is intact, but they are in the log file in chronological order, not grouped by request. That is fine. You can filter on a request UUID if you want to follow a single request in isolation.
Even if a write was atomic, maybe another process actually winds up writing at the offset the original process is about to and it winds up overwriting.
Don't allow multiple processes to write to the same file or log. Use process.stdout and all will be fine. Or if you really want to log directly to the filesystem, use an exclusive lock mechanism.
What is the best practice for writing to log files in NodeJS while ensuring atomicitiy?
process.stdout, one write call per coherent log message. You can let your process supervisor (systemd or upstart) write your logs for you, or use a log manager such as multilog, sysvlogd and pipe your stdout to them and let them handle writing to disk.
I am building a simple application to download a set of XML files and parse them into a database using the async module (https://npmjs.org/package/node-async) for flow control. The overall flow is as follows:
Download list of datasets from API (single Request call)
Download metadata for each dataset to get link to XML file (async.each)
Download XML for each dataset (async.parallel)
Parse XML for each dataset into JSON objects (async.parallel)
Save each JSON object to a database (async.each)
In effect, for each dataset there is a parent process (2) which sets of a series of asynchronous child processes (3, 4, 5). The challenge that I am facing is that, because so many parent processes fire before all of the children of a particular process are complete, child processes seem to be getting queued up in the event loop, and it takes a long time for all of the child processes for a particular parent process to resolve and allow garbage collection to clean everything up. The result of this is that even though the program doesn't appear to have any memory leaks, memory usage is still too high, ultimately crashing the program.
One solution which worked was to make some of the child processes synchronous so that they can be grouped together in the event loop. However, I have also seen an alternative solution discussed here: https://groups.google.com/forum/#!topic/nodejs/Xp4htMTfvYY, which pushes parent processes into a queue and only allows a certain number to be running at once. My question then is does anyone know of a more robust module for handling this type of queueing, or any other viable alternative for handling this kind of flow control. I have been searching but so far no luck.
Thanks.
I decided to post this as an answer:
Don't launch all of the processes at once. Let the callback of one request launch the next one. The overall work is still asynchronous, but each request gets run in series. You can then pool up a certain number of the connections to be running simultaneously to maximize I/O throughput. Look at async.eachLimit and replace each of your async.each examples with it.
Your async.parallel calls may be causing issues as well.
I am trying to write a Node.js program to execute and monitor javascript programs. I am looking for a way to find out whether the monitored program is still "running" i.e. doing anything useful.
In my current approach, when receiving code to test, I start a new child process and hand the code to it. The child process instruments the code creates a Sandbox using Contextify and executes the code using this sandbox.
After the sandbox.run(code) call returned I know that the blocking part of the code finished and can show that in the UI. However, I don't now whether the code registered any timers using setTimeouts or created any other event sources that would cause parts of the code to be exited later. So I don't know whether it's really "finished" yet.
Is there a way in Node.js to check whether there are still events on the event loop to be handled (or even better, how many are left)?
I found this other question, but it only talks about how to monitor the event loop to find out whether the performance of node is still fine. But I'm not interested in the performance (I don't care if the executing code is blocking for 10s or only doing something for 1ms every 2 minutes) and I don't want to use outside tools but find out about the state of the event loop from inside node itself. Is that possible?
I solved my problem in a way, although I didn't find a general answer to the question.
The idea here is that the process will exit by itself if it has executed all the code it was started with and no EventEmitters are registered anymore. This was basically what I wanted, since I wanted to be notified when the process was "done" and now I could just listen to the "exit" event of the child_process.
But my process that executed the code didn't exit by itself. This had two reasons:
I used a timer to regularly send the data gathered about the execution to the parent process. If such a timer is registered the process won't exit. You could unref the timer but I was afraid that would lead to data loss since the process could quit before the last bit of data was sent out (since it wouldn't wait for the timer to execute). So I changed my code to only schedule a timer if there was data to be sent out instead of regularly checking for data to be sent.
I used fork to create the child process. This also creates a communication channel between the parent and child process and since I needed to send the code to execute to the child_process I registered the child process for messages received from the parent using process.on("message", callback). However, now we have another EventEmitter registered which prevents the process from quitting. Luckily, I realized that I only needed one message from the parent process and no further messages so I could remove the event emitter after receiving that message. So instead of process.on() I used process.once() which will execute the callback only once and automatically remove the event emitter after that. Exactly what I wanted. Alternatively you could use process.removeListener().
Now I just wait for the child_process to exit and thus know that everything is finished and can notify the client.
So the solution here is to make sure none of your own EventEmitters keep the process alive and then just wait for it to exit.
I've gathered that due to the event queue design of node.js and the way that JavaScript references work, that there's not an effective way of figuring out which active request object (if any) spawned the currently-executing function.
If I am in fact mistaken, how do I maintain some sort of statically-accessible context object that is available for the life of a request, short of manually passing it around every function in the event/callback chain?
If there's no way to do that, is there a way to somehow wrap every event call so that when it is spawned, it copies the context data from the event that spawned it?
What I'm actually trying to achieve is having every log message also contain the username of the person to whom the log message is related. Because the callback event chain can become quite complex, I'd say trying to pass the request object around every function in the application is a bit messy for my taste, so hopefully there's another way.