I am making a certain client -> server application using CherryPy as the web server.
I will be needing to create a request with a large content-length header while sending about 80% of the size of the content but then i don't want CherryPy to read the post data based on the content-length i sent, i want to read it manually and write to another file. but it seems CherryPy times out waiting for whole content-length.
In other words i want to read the incoming post stream manually but still allow CherryPy to process the request headers (and not the body)
UPDATE: I think I can do this with a 'Custom Processor' : http://docs.cherrypy.org/stable/refman/_cpreqbody.html , but i still don't understand how i can write a processor and call it in my application.
You could try doing it with rfile, but see the warning. You should really look for a solution that doesn't break the standards. Perhaps use WebSocket.
Related
I need to make an WEB based monitoring system using ESP8266, which could display the data. The system will have a user registration form, which should allow to display the data for a particular user. For this purpose I got a remote server (domain). Now I'm facing with some problems, how could I send data to this domain from the ESP? My ESP module uses NodeMCU firmware and I can program it using Lua. I read that there is HTTP GET and POST request methods and I unsuccessfully spent a few days trying to implement one of these methods... Maybe someone could put me on the road What should be the sequence of steps to start sending data to the external server? That would be a big step forward if I could send f.e. constant value variable.
Assuming your NodeMCU is connected to a network and had internet access, you can just do
http.post(url, headers, body, callback)
and it should send a post request to the given URL. HTTPS also works here, but has limitations.
Note that you need to compile the firmware with the HTTP (and TLS if you want HTTPS) module(s) by uncommenting the corresponding line(s) in the app/include/user_modules.h file.
I was doing some nodejs and I ran into a scenario in which I had to use POST requests. I saw that node deals with POST requests in a slightly different manner than the GET requests. In the case of POST requests we need to create two event listeners on('data', ...) and on('end', ...) . In the case of GET requests, I found no such complication. All of this led me to believe that maybe GET requests are always guaranteed to be sent within one chunk of data from the client. Whereas, POST requests can be sent over multiple chunks. Am I correct, or is there any flaw in my understanding. Please correct me if so.
GET requests don't usually have a "body" as part of them so once you've read the http request headers, you have everything so there is no need for additional code to read more.
POST requests, on the other hand, usually do have a body so once you've gotten the headers, you then need to read the body.
FYI, TCP is a streaming protocol which means there are no guarantees about what chunks data will arrive in. Even the headers themselves could arrive in multiple packets. But, the http library you're using already takes care of that for you. It reads data until it has all the headers. Reading the body of a POST request is more up to you to do unless you use some sort of body-parser middleware which will read the body for you.
Is there a way, I could continuously send a response to the client before finally calling end on the response object?
There is a huge file and I want to read,parse and then send the analysis back to the client. Is it possible that I keep sending a response as soon as I read a line?
response.send also calls end, so I cannot use it.
You want to use res.write, which sends response body content without terminating the response:
This sends a chunk of the response body. This method may be called multiple times to provide successive parts of the body.
Note that write is not included in the Express documentation because it is not an Express-specific function, but is inherited from Node's core http.ServerResponse, which Express's response object extends from:
The res object is an enhanced version of Node's own response object and supports all built-in fields and methods.
However, working with streaming data is always little tricky (see the comment below warning about unexpected timeouts), so it may be easier to restructure your application to send streaming data via a WebSocket, or socket.io for compatibility with browsers that don't support WebSockets.
What kind of server side application are you working with? You could use Node.js with Socket.IO to do something like this. Here's some useful links.
First Node.js application
Node.js readline API
i am just now digging into everything node.js related as i am tasked with a somewhat complex requirement. I need to implement a kind of a proxy service using node.js, i am not even sure it can be called a proxy per-se.
What i need is the following:
setup a web server that would listen to requests, then, query a database for a list of URLs (according to some data from the original request), curl these URLs and according to the responses from these URLs construct a response to the original client. The response would be some kind of mix and mash from these multiple responses according to some business logic.
So in short :
Client -> NODE -> NODE queries DB, calls N Urls, processes responses, construct response to client -> return response to client.
I've looked at node-http-proxy but it doesn't seem like a good match for what i need.
Basically i am looking at pointers of how can this be accomplished. Performance is of outmost importance as we'll be serving 50K-100K QPS or more.
I don't think that you will find a module that will do the job for you. You have to write something completely custom. You can pick a framework like express, that will create the server and will handle the requests for you. Then pick the database driver that you need to connect to the database and use the request module to fire the requests to the URLs that you will retrieve from the database. It can be a quite an overhead if you have to fire a lot of requests. Async module and specifically async.each will help you. This can fire all the requests in parallel and at the very last callback have the results in an array. Process the results, send them back to the client. It should not be quite complicated.
In a node.js server that accepts HTTPS post requests that are typically pretty large (a few MBs) we want to be able to start processing the requests before the entire thing is accepted by the server.
For example, if a request with a big fat body arrives, we want to look at its path and based on it decide whether to terminate/reject it, without having to wait for the entire request to arrive (and pay IO cost of receiving that fat body).
You could try the the Connect Limit middleware:
https://github.com/senchalabs/connect/blob/master/lib/middleware/limit.js
or, implement your own solution in a similar way by checking req.headers[content-length], etc..
Based on experimentation, it seems that Node.js only fires the request event after parsing the HTTP headers. Meaning there's a chance to examine the headers before we even start listening for the data event.
Thus the solution seems to be to check the headers before reading any data, and potentially rejecting the request at that point. If we don't reject at that point, we start accumulating the data buffers as they arrive and if they exceed the limit (and thus conflict with the reported content length) we have another chance to reject the request right there by calling response.end().