var fileStream = fs.createReadStream(filePath)
how to have readStream with shared/exclusive lock
so, that file can not be deleted or altered
I don't think node exposes any filesystem locking mechanisms.
If you were going to use filesystem for system-wide locks or secure inter-process communication, you'll need to find another way (e.g. sockets).
If it's not security critical there are some ways of making it harder (but not impossible) for other processes to mess with your files:
Use unguessable filenames. require('crypto').getRandomBytes('16').toString('hex')
Narrow permissions when creating files via options on createReadStream.
Run node process as a special user, so files will be owned only by that user. Either configure OS to run node under appropriate user, or have node run as root and switch to another user via process.setuid/setgid.
Related
I'm struggle with cashed sftp session factory. Namely, I suffered from session unavailable because I used to many in my application. Currently I have one default non cashed session. Which writes file to sftp server but set up locks on them. Thus it can't be read by any other user. I'd like to avoid it. Perfectly, turn off lock after single file is uploaded. Is it possible ?
Test structure
Start polling adapter
Upload file to remote
Check whether files are uploaded
Stop polling adapter
Clean up remote
When you deal with data transferring over the network, you need to be sure that you release resources you use do to that. For example be sure to close InputStream after sending data to the SFTP. This is really not a framework responsibility to close it automatically. More over you may give us already not an InputStream, but just plain byte[] from it. That's only a reason I can think about locking-like behavior.
My express server does a very simple work - saving the request url into a file (via fs.appendFile).
I suppose it works fine when not using pm2, because it has only one process, so no other process/thread saving the same file at the same time.
But when using pm2, I don't know if it will occur two processes write the same file at the same time? Thanks.
When you use pm2 on cluster mode, request routing will happen
using Round Robin algorithm. That means the cluster master accepts
all the incoming connections and routes them to the child processes
(one request to one child process).
So, one request will be routed to one child process and the same request won't be processed by another process.
For your above case, When you receive two different requests from two different clients then they will be processed by two different processes.
As long as you have a logic to create a unique file name even though the requests handled at the same time, you won't get any issues.
You will get issues only if you try to write the files by two different processes with the same file name.
If you write different files from different clients with different file names then it won't be an issue.
Note: As a request from one client will be processed by one process, two or more processes won't process the same request and won't write the same file twice.
The issue will occur, If you write different files from different clients with the same file name.
Hope you understand :-)
Yes it may mess when multiple processes writing/appending the same file at the same time. So then best way is to only use one process to write file, or you have to synchronize them
I have to serve a calculation via algorithm, I've been advised to use a child process per each opened socket, what I am about to do is something like that:
var spawn = require('child_process').spawn;
var child = spawn('node', ['algorithem.js']);
I know how to send argument to the algorithm process and how to receive results.
What I am concerned about, is how many socket (each socket will spawn a process) I can have?
How can I resolve this with my cloud hosting provider? so that my app gets auto scaled?
What's the recommended node js cloud hosting provider?
Finally, is this a good approach in using child processes?
Yes, this is a fair approach when you have to do some heavy processing in node. However, starting a new process introduces some overhead, so be aware. The number of sockets (file descriptors) you can open is limited by your operating system. On Linux, the limits can seen using for example the ulimit-utility.
One alternative approach, that would remove the number of sockets/processes worry, is to run a separate algorithm/computation-server. This server could spawn N worker threads and would listen on a socket. When a computation request is received, this can for example be queued and processed by the first available thread. An advantage of this approach is that your computation server can run on any machine, freeing up resources for your node instance.
The Apache Commons FTPClient creates calls openDataConnection everytime i.e for every command it uses a separated socket.
Which means many ports are used for data transfer? Because of this sometimes i am getting into SockeReadTimeOutException which results because some Timed_Waiting ports are being used.
Not able to understand why don't a single port used for data transfer.Which consumes less memory and less stress on system. Any Advice??
If this aspect is important to you, you may search another library. If your system allows secured file transfer (SFTP), have a look to: JSch.
I did not check the code but it might work differently compared to FTPClient and might not open a socket for each command.
I am designing a server for the following scenario:
a series of single images are stored on a NAS, lets say 100 of them
a client connects to the server over TCP socket and requests image39
server reads image39 from NAS and sends back to client over socket
it is likely that the client will also request other images from the series, so:
I would like to launch a thread that iterates through the images, reads them, and does a cat image39 > /dev/null to force cache into memory on server
thread will fetch images as follows: image38, image40, image37, image41, etc.
already fetched images are ignored
if client now requests image77, I want to reset the fetch thread to fetch: image76, image78, etc.
This has to scale to many series and clients. Probably on the order of 1000 concurrent
prefetches. I understand that threads can cause performance hit if there are too many. Would it be better to fork a new process instead? Is there a more efficient way than threads or processes ?
Thanks!!!
This is premature optimization. Try implementing your system without tricks to "force" the cache, and see how it works. I bet it'll be fine--and you won't then need to worry about nasty surprises if it turns out your tricks don't play nice with other things on the system.