Scrapy - is it possible to extract Payload Request from Response - python-3.x

is it possible to extract and set as variable the "Payload Request" which has been pushed in order to receive particular response?

You can access the request object in the callback function by response.request.
This object is the request object itself, so it contains everything you passed in the request. It doesn't have a "payload" attribute though.
The equivalent should be response.request.body, assuming you had a body in the request. Everything else is still there, headers, cookies, meta, method, etc
More on the params of request here.

Related

request and respond in function. Nodejs

If I change position of req with res for example (res,req)=>{ res.send();}
the server shows error that res.send is not a function.
My question is are we restricted with positioning the req,res objects?
why it throws the error the res.send() is not a function or why is it trying to see it as a function?
(res,req) => {res.send();}
is a function with two parameters. When it is invoked by the express framework, the first will be passed a request object (which has no .send function) and the second will be passed a response object (which has a .send function). Your code wants to invoke the .send function of the first object (which you misleadingly call res), but that first object is a request and has no .send function.
See the express API documentation here.
You can change the variable name, can push new variables to request but you can not change functionality of internal request and response object. And as per request functionality , it accepts what you send in request and Send() is not part of request.

Why is the request body blank when the content-type is application/x-www-form-urlencoded?

I am receiving a request with content-type application/x-www-form-urlencoded. When I try to read the body of the request using cherrypy.request.body.read() the results is b''.
I seem to be able to access the request form parameters using any of these:
cherrypy.request.params
cherrypy.request.body.params
cherrypy.request.body.request_params
But this is inconvenient for my use case, I want to be able to obtain the raw request body regardless of content-type. Also the above 3 give me a dictionary, which isn't the exact format that the request had in its body. Is there a way to do that with cherrypy? Or is this functionality hidden?
Not sure what are you trying to accomplish by not using the already parsed body that correspond to the defined Content-Type... but you can process the body of the request yourself configuring: cherrypy.request.process_request_body = False and read the body with something like:
cherrypy.request.rfile.read(cherrypy.request.headers['Content-Length'])
For more information see: https://github.com/cherrypy/cherrypy/blob/master/cherrypy/_cprequest.py#L292-L315
Fragment of relevant parts of that url:
rfile = None
"""
If the request included an entity (body), it will be available
as a stream in this attribute. However, the rfile will normally
be read for you between the 'before_request_body' hook and the
'before_handler' hook, and the resulting string is placed into
either request.params or the request.body attribute.
You may disable the automatic consumption of the rfile by setting
request.process_request_body to False, either in config for the desired
path, or in an 'on_start_resource' or 'before_request_body' hook.
WARNING: In almost every case, you should not attempt to read from the
rfile stream after CherryPy's automatic mechanism has read it. If you
turn off the automatic parsing of rfile, you should read exactly the
number of bytes specified in request.headers['Content-Length'].
Ignoring either of these warnings may result in a hung request thread
or in corruption of the next (pipelined) request.
"""
process_request_body = True
"""
If True, the rfile (if any) is automatically read and parsed,
and the result placed into request.params or request.body.
"""
body = None
"""
If the request Content-Type is 'application/x-www-form-urlencoded'
or multipart, this will be None. Otherwise, this will be an instance
of :class:`RequestBody<cherrypy._cpreqbody.RequestBody>` (which you
can .read()); this value is set between the 'before_request_body' and
'before_handler' hooks (assuming that process_request_body is True."""

What does "content-type" mean in headers of python requests library and if the value is text/html;charset=UTF-8?

I want to do some operations with response from python requests library. After I use below function;
response = requests.get(f'{AUTHORIZE_URL}?client_id={CLIENT_ID}&response_type=code&state={STATE}&redirect_uri={REDIRECT_URI}')
I need to get an URL something like this in return;
http://127.0.0.1:8000/products/auth/?state=2b33fdd45jbevd6nam&code=MGY1MTMyNWY0YjQ0MzEwNmMxMjY2ZjcwMWE2MWY5ZDE5MzJlMjA1YjdkNWExNGRhYjIzOGI5NzQ5OWZkNTA5NA
While doing it, it will be easier to use JSON in order to get state and code values from URL but I cannot use it because I think the content type does not allow this.
See this for Content-Type explanation: Content-Type
In short the "content-type" in the headers of response got by using requests.get tells you what kind of the content server did send, in your case you'we got a response in the form of the HTML (like .html document) and you can read that response with response.text, if the "content-type" is "application/json" then you can read it as JSON like this response.json().
I see that you use some local server, your local server should send in headers "Content-Type": "application/json" and then you should be able to read JSON from response like this (you need to send JSON not hmtl or text from server):
targetURL = 'http://127.0.0.1:8000/products/auth/?state=2b33fdd45jbevd6nam&code=MGY1MTMyNWY0YjQ0MzEwNmMxMjY2ZjcwMWE2MWY5ZDE5MzJlMjA1YjdkNWExNGRhYjIzOGI5NzQ5OWZkNTA5NA'
response.get(targetURL).json()

what is the difference between ServerResponse and http.IncomingMessage?

what is the difference between ServerResponse and http.IncomingMessage? I'm new in Node,and I'm really confused about these two object.
The http.IncomingMessage object represents the request information from the client received by the server. See this link for the Node API docs. So, this object contains things like the headers, the method (GET, POST, etc...), the URL requested, etc... It implements the ReadableStream interface -- which means you can read data from it using a stream -- and has several events, methods and properties. (too many to list here...)
The ServerResponse object represents the http server's response to the client. See this link for the Node API docs. With this object, you can set the headers that will be sent to the client, the status code and status message of the response, as well as any data to be returned. It implements the WritableStream interface -- which means you can write to it using a stream -- and also has several events, methods, and properties.
In many server side programs you'll see something like:
function foobar(req, res) {
}
The req variable holds the http.IncomingMessage object -- it's short for request. The res variable holds the ServerResponse object -- it's short for response.
Please see the documentation links for a full explanation of both objects.
Hope this helps!
req is an IncomingMessage object and res is a ServerResponse object.
So check for unique properties on each, for example if the particular object has a writeHead() function, then it's the response object.
You may also be able to use instanceof to check: res instanceof http.ServerResponse.

node-rest-client POST Vs. PUT

Has anyone used node-rest-client? For POST and PUT methods, it says "To send data to remote site using POST or PUT methods, just add a data attribute to args object:" So how can I distinguish between a POST call and PUT call?
To make a put request, you just use the put method instead of the post or get methods.
client.put("http://remote.site/rest/xml/method", function(data, response){
// parsed response body as js object
console.log(data);
// raw response
console.log(response);
});

Resources