Not getting response from AWS after uploading image through ESP32 - node.js

General context:
I am working on an IoT application where I upload images from an ESP32 connected to an SBC.
The uploading is done through an API provided by a third-party backend developer.
The upload API works through other mediums (such as Postman, python requests library, python http client library)
The ESP32 is connected to the SBC through UART.
I construct/generate the HTTP request on the SBC and send it as bytes. I have written a function on ESP32 that can send the bytes as a generic HTTP request, to the URL specified.
Then it sends the response string back to the SBC.
All of this works. For small requests, I am facing no issues. I am able to download images, etc.
However, when uploading an image, I don't get a response and I end up timing out after 30s. I checked without timeout FYI, but no response.
I checked from the server-side. It appears my request has succeeded and the server is sending me 200 with the URL of the image. Using that URL, I was able to verify that the image was uploaded successfully.
However, I do not receive this response on the microcontroller.
Not sure what the issue is. Any suggestions as to what I can do?
I can't give out the code but I'll send a general structure:
ESP32
-> Receives URL, port, length of request
-> Connects to server and reads the request from UART and writes to server
-> Wait for response after response is sent
Python raw http
POST (server path) HTTP/1.1
Host: (url)
correlation-id: test5
Content-Type: multipart/form-data; boundary=WebKitFormBoundary7MA4YWxkTrZu0gW
Authorization: Bearer (access token)
Content-Length: 268
--WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name="portraits"; filename="name"
Content-Type: image/jpeg
(data)
--WebKitFormBoundary7MA4YWxkTrZu0gW--
Edit 1:
So, turns out it is not only "upload image", some other requests are also behaving similarly. Our server has many microservices. The services written in nodeJS which have more than 1 redirects are not working...?

I figured out what the issue is and hopefully, it will help anyone else facing the same issue. Also, some of my requests to the backend server which used a different authentication method worked
I had been generating the raw HTTP using postman code generation but it turns out Postman doesn't add a few headers which are needed for communicating with more complex servers.
What I mean is that if I host a local server, the above code will work. I had already tested it that way
What solved my problem is adding these headers:
POST (server path) HTTP/1.1
Host: (server URL)
User-Agent: ESP32
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
correlation-id: test
Authorization: Bearer (access_token)
Content-Length: 146360
Content-Type: multipart/form-data; boundary=af59ef02d60cd0efefb7bc03db1f4ffc
--af59ef02d60cd0efefb7bc03db1f4ffc
Content-Disposition: form-data; name="portraits"; filename="(name)"
Content-Type: image/jpeg
(data)
--af59ef02d60cd0efefb7bc03db1f4ffc--

Related

Azure Logic App - Receive file from http request

I have an ASP.Net handler that returns a PDF report. I want the Azure Logic App to request the file and then add it to an e-mail as an attachment.
When I try to do this through an HTTP request I get the following error:
BadRequest. Http request failed as there is an error: 'Error while copying content to a stream.'
If I make the request with a browser I get a HTTP 200 response and it works. See request/response headers from chrome and fiddler.
I'm sure I could solve this with an Azure Function to get the file blob and pass it to the e-mail stage but it appears in the documentation that Logic Apps can handle streams and base64 encode. Am I missing something here?
I tried with the following a static result in an HTTP request to mimic the HTTP request/stream as much as possible. I guess it comes down to that you need to design the body of the stream in a way the includes content and content-type like I did below with my mockup HTTP request
pdf content
content-type: application/pdf and application/octet-stream worked
Send and email action:
Sent email
Outlook result:

How to capture what Alexa console sends to the endpoint?

I am currently learning about the chatbots and developed two different skills. One has an endpoint on Azure and the other's endpoint is on AWS. Everything works fine. I would like to know if there is a way to see what the Alexa console sends to your endpoint (including the header)? The Alexa console only shows the body. I would like to capture header and body so I can test my endpoints with Postman.
POST / HTTP/1.1
Content-Type : application/json;charset=UTF-8
Host : your.application.endpoint
Content-Length :
Accept : application/json
Accept-Charset : utf-8
Signature:
SignatureCertChainUrl:
https://s3.amazonaws.com/echo.api/echo-api-cert.pem
It's the HTTP HEADER that Alexa is sending with its Body
For more info : Request and Response JSON Reference

Receiving webhook in express from local application

I have an application (headless CMS) running locally. It has an option to send a webhook to another application. I have been trying to interpret this webhook via express using posts. I have not been able to even get it to register a request coming in from the application. I tested this route using postman and found that it is working when I post to it.
router.post('/', (req, res) => {
console.log("Recieved");
console.log(req.body);
res.status(200).send('ok')
});
Thus, when I send a post to: http://localhost:3000/recall via postman. I get the following header back:
Access-Control-Allow-Credentials →true
Access-Control-Allow-Headers →X-Requested-With,content-type
Access-Control-Allow-Methods →GET, POST, OPTIONS, PUT, PATCH, DELETE
Access-Control-Allow-Origin →http://localhost:3000
Connection →keep-alive
Content-Length →2
Content-Type →text/html; charset=utf-8
Date →Mon, 03 Sep 2018 21:23:22 GMT
ETag →W/"2-eoX0dku9ba8cNUXvu/DyeabcC+s"
X-Powered-By →Express
With the body:
ok
My script also prints the body of the post.
I can verify the webhook is working by testing it with request bin. I get back the following:
FORM/POST PARAMETERS
None
HEADERS
Cloudfront-Forwarded-Proto: http
Cloudfront-Is-Mobile-Viewer: false
Cloudfront-Is-Desktop-Viewer: true
Connect-Time: 1
Via: 1.1 3566cbcd49f71967b52a565888e4d272.cloudfront.net (CloudFront), 1.1 vegur
Content-Length: 387
Connection: close
Accept: */*
Content-Type: application/json
Cloudfront-Viewer-Country: US
X-Amz-Cf-Id: dRe5CvkLFJZJNcpZbhmeEHo0ar_taj6guvN8utwkyVXM7ZMJc5BZTw==
Cloudfront-Is-Smarttv-Viewer: false
X-Request-Id: 4b6d2cdc-5c45-495b-b358-2e808e1bfeb4
Cloudfront-Is-Tablet-Viewer: false
Total-Route-Time: 0
Host: requestbin.fullcontact.com
BODY
{"event":"singleton.remove","hook":"Save After Sington","backend":1,"args":[{"name":"Wonder","label":"Wonder","_id":"Wonder5b8cef36a0097","fields":[{"name":"Best","label":"","type":"text","default":"","info":"","group":"","localize":false,"options":[],"width":"1-1","lst":true,"acl":[]}],"template":"","data":null,"_created":1535962934,"_modified":1535962934,"description":"","acl":[]}]}
I tried enabling cross-origin requests. How could I fix this problem? My thought is it has something to do with the fact this request is originating and ending locally.
For an application to consume webhooks it needs to have a publicly accessible URL. Basically, the rest of the world (internet) doesn't know your localhost:3000 endpoints exist.
An easy way to fix this is to use a lightweight tool like ngrok to expose your local server; in turn allowing other applications to communicate with yours.
You will need to define the specific callback route that you want to consume the webhook POST request. Examples below.
Run your node script
Turn on ngrok
send webhook POST requests to your endpoint using the NGROK https address
Now, instead of sending your webhook to localhost:8000/MyWebhookConsumingEndpoint
you send it to
https://95e26af4.ngrok.io/MyWebhookConsumingEndpoint

How to ensure that webhook is receiving data from desired data source and not third party?

I'm currently building out an application that utilizes Shippo's webhooks functionality, but wondering if it's possible to ensure that the HTTP POST requests are coming from Shippo's servers rather than some malicious third party.
Adding a webhook to Shippo:
https://shippo-static.s3.amazonaws.com/img/illustrations/webhooks.png
When recieving HTTP requests, it appears as though Shippo doesn't provide a key or anything that I can use on my end to verify that the request came from shippo themselves. Is there any way that I would be able to retroactively and securely ensure that the request came from Shippo's servers?
Sample headers:
User-Agent: python-requests/2.9.1
Via: 1.1 vegur
Accept: /
Connection: close
Content-Length: 2203
Total-Route-Time: 0
Cf-Connecting-Ip: 54.87.248.176
Cf-Ipcountry: US
X-Request-Id: cec0a1aa-6a1a-47e9-ac9d-c685a893591d
Cf-Ray: 3d0718455e149fea-IAD
Connect-Time: 1
Accept-Encoding: gzip
Host: requestb.in
Shippo-Api-Version: 2017-08-01
Cf-Visitor: {"scheme":"https"}
Content-Type: application/json
Sample JSON Response:
https://goshippo.com/docs/tracking
You need to do two things:
Have your webhook include some sort of token, i.e. have shippo call https://yourapp.com/webhookroute/?secure_token=123abc. Parse and check the token server side. Can add multiple tokens, i.e. https://yourapp.com/webhookroute/?secure_token_1=123abc&secure_token_2=456def
Use SSL/TLS certs when deploying your app, which will encrypt the URL/tokens when shippo sends the webhook. Make sure the webhook calls https:// [rest of url], not http://. This will hide the tokens from the rest of the internet.
If the tokens are set as env vars server side, this should be secure.

Node.js: Why does post request w/ png have content-length header value that is smaller than the actual file size?

I'm making a post request to a local node.js server. The post request contains a .png file that is 2.6 kB. The post request is issued using curl:
curl -v -X POST -H "Content-Type: image/png" -d #node.png http://localhost:3000
The request output of the curl command is:
POST / HTTP/1.1
User-Agent: curl/7.38.0
Host: localhost:3000
Accept: */*
Content-Type: image/png
Content-Length: 1328
Expect: 100-continue
Note that the Content-Length header is showing up as 1328 Bytes which is about half of the actual file size being posted. The content-length should be around 2.6 kB, correct? Thanks for your help!
It is actually request header for downloading a partial file.
please check out this from Wikipedia,
The server has received the request headers and the client should
proceed to send the request body (in the case of a request for which a
body needs to be sent; for example, a POST request). Sending a large
request body to a server after a request has been rejected for
inappropriate headers would be inefficient. To have a server check the
request's headers, a client must send Expect: 100-continue as a header
in its initial request and receive a 100 Continue status code in
response before sending the body. The response 417 Expectation Failed
indicates the request should not be continued.
Wiki Link

Resources