recreating cURL request with -I, -H flags in nodeJS - node.js

On the command line, I can do a request like: curl -I -H "Fastly-Debug: 1"
and it will return a lot of helpful information from the CDN serving that URL, in this case, Fastly:
cache-control: public, max-age=0, must-revalidate
last-modified: Tue, 20 Apr 2021 21:17:46 GMT
etag: "4c5cb3eb0ddb001584dad329b8727a9a"
content-type: text/html
server: AmazonS3
surrogate-key: /v3.6/tutorial/nav/alerts-and-monitoring/
accept-ranges: bytes
date: Fri, 30 Apr 2021 20:50:15 GMT
via: 1.1 varnish
age: 0
fastly-debug-path: (D cache-lga21923-LGA 1619815815) (F cache-lga21940-LGA 1619815815)
fastly-debug-ttl: (M cache-lga21923-LGA - - 0)
fastly-debug-digest: 04c3e527819b6a877de6577f7461e132b97665100a63ca8f667d87d049092233
x-served-by: cache-lga21923-LGA
x-cache: MISS
x-cache-hits: 0
x-timer: S1619815815.944515,VS0,VE136
vary: Accept-Encoding
content-length: 65489
How can I do this in node?
This is my attempt:
const headers = {
'Fastly-Key': environment.getFastlyToken(),
'Accept': 'application/json',
'Content-Type': 'application/json',
'Fastly-Debug': 1
};
async retrieveSurrogateKey(url) {
console.log("this is the url: ", url)
try {
request({
method: `GET`,
url: url,
headers: headers,
}, function(err, response, body) {
if (err){
console.trace(err)
}
console.log(request.headers)
})
} catch (error) {
console.log("error in retrieval: ", error)
}
}
Is there a way for me to pass in the -I and -H flags?

The -H (header) flag allows you to specify a custom header in cURL. Your code already does this - great! All that's left is emulating the -I (head) flag.
From cURL manpage for the -I option:
Fetch the headers only! HTTP-servers feature the command HEAD which this uses to get nothing but the header of a document.
To use the HEAD method, you need to specify it instead of GET:
method: `HEAD`
And finally, the headers returned by the server in the response can be obtained from response.headers.

Related

Curl HEAD request returns a last-modified header but Node https.request(..., {method: HEAD},...) does not

Apparently there's something I don't know about HEAD requests.
Here's the URL: 'https://theweekinchess.com/assets/files/pgn/eurbli22.pgn', which I'll refer to as <URL> below.
If I curl this, I see a last-modified entry in the headers:
curl --head <URL>
HTTP/2 200
last-modified: Sun, 18 Dec 2022 18:07:16 GMT
accept-ranges: bytes
content-length: 1888745
host-header: c2hhcmVkLmJsdWVob3N0LmNvbQ==
content-type: application/x-chess-pgn
date: Wed, 11 Jan 2023 23:09:14 GMT
server: Apache
But if I make a HEAD request in Node using https, That information is missing:
https.request(<URL>, { method: 'HEAD' }, res => {
console.log([<URL>, res.headers])}).end()
This returns:
[
<URL>
{
date: 'Wed, 11 Jan 2023 23:16:15 GMT',
server: 'Apache',
p3p: 'CP="NOI NID ADMa OUR IND UNI COM NAV"',
'cache-control': 'private, must-revalidate',
'set-cookie': [
'evof3sqa=4b412b5913b38669fc928a0cca9870e4; path=/; secure; HttpOnly'
],
upgrade: 'h2,h2c',
connection: 'Upgrade, Keep-Alive',
'host-header': 'c2hhcmVkLmJsdWVob3N0LmNvbQ==',
'keep-alive': 'timeout=5, max=75',
'content-type': 'text/html; charset=UTF-8'
}
]
I tried axios instead of https:
const response = await axios.head('https://theweekinchess.com/assets/files/pgn/eurbli22.pgn');
console.log({response: response.headers})
And that works (incl. the proper MIME type):
date: 'Thu, 12 Jan 2023 20:00:48 GMT',
server: 'Apache',
upgrade: 'h2,h2c',
connection: 'Upgrade, Keep-Alive',
'last-modified': 'Sun, 18 Dec 2022 18:07:16 GMT',
'accept-ranges': 'bytes',
'content-length': '1888745',
'host-header': 'c2hhcmVkLmJsdWVob3N0LmNvbQ==',
'keep-alive': 'timeout=5, max=75',
'content-type': 'application/x-chess-pgn'
I also tried waiting for re2.on('end', console.log(res.headers)), but same output as before.
I'm going to close this issue and post it instead as a 'bug' on Node's site. I'm sure there's something that needs to be changed in how I'm executing the HEAD request.

Finding the final URL that the request will be redirected to in nodeJS?

I have the following URL:
https://forecast.weather.gov/zipcity.php?inputstring=95014
I would like to figure out which URL it will redirect to. In this example it is:
https://forecast.weather.gov/MapClick.php?CityName=Cupertino&state=CA&site=MTR&lat=37.3042&lon=-122.095
I tried multiple solutions such as:
res.headers.location
and
res.headers.get('location')
But none of them seem to work. I know that the URL redirects because, I could successfully redirect in curl and google-chrome. Here is the code that I am running:
https.get('https://forecast.weather.gov/zipcity.php?inputstring=95014', res => console.log(res.headers.location))
When I was running in curl, I ran the following:
curl -Ls -o /dev/null -w %{url_effective} http://forecast.weather.gov/zipcity.php?inputstring=95014
And got the desired output of:
http://forecast.weather.gov/MapClick.php?CityName=Cupertino&state=CA&site=MTR&lat=37.3042&lon=-122.095
When I run curl:
$ curl -I https://forecast.weather.gov/zipcity.php?inputstring=95014
HTTP/2 302
server: Apache
x-nids-serverid: www1.mo
location: https://forecast.weather.gov/MapClick.php?CityName=Cupertino&state=CA&site=MTR&lat=37.3042&lon=-122.095
x-ua-compatible: IE=Edge
access-control-allow-origin: *
content-type: text/html; charset=UTF-8
content-length: 0
cache-control: max-age=722
expires: Mon, 26 Apr 2021 19:28:03 GMT
date: Mon, 26 Apr 2021 19:16:01 GMT
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload
As you can see, there are multiple headers present here that are not present inside nodejs's results:
{
server: 'AkamaiGHost',
'mime-version': '1.0',
'content-type': 'text/html',
'content-length': '288',
expires: 'Mon, 26 Apr 2021 19:21:50 GMT',
date: 'Mon, 26 Apr 2021 19:21:50 GMT',
connection: 'close',
'strict-transport-security': 'max-age=31536000 ; includeSubDomains ; preload'
}
I would like to stick to the http module.

Has anyone an idea about IBM OpenWhisk and Cors settings?

i have the following problem. When run my openWhisk function through the API gateway all is fine. Also calling it as a cross domain request works well.
But i need to call it with credentials and it seems there is no option to do this.
What i tried:
just enabling the API CORS setup without any headers in the function itself
enabling cors in the API und having CORS headers in the function as well ->the relevant headers (access-origin) in the function get overwritten!
disabling CORS in the API and having CORS headers in the function -> the relevant headers (access-origin) of the function get deleted
This is the code that normally should work:
return {
headers: {
'Access-Control-Allow-Headers': '*',
'Access-Control-Allow-Origin': domain,
'Access-Control-Allow-Credentials': 'true',
'Content-Type': 'text/xml'
},
body: xml
}
Would be great if anyone has an idea because the support is not answering my ticket for many days now.
Thanks and best, André
Using your sample:
> cat t.js
function main() {
return {
headers: {
'Access-Control-Allow-Headers': '*',
'Access-Control-Allow-Origin': 'domain',
'Access-Control-Allow-Credentials': 'true',
'Content-Type': 'text/xml'
},
body: "<hi></hi>"
}
}
Against Bluemix:
> wsk action update t t.js -a web-custom-options true --web true
ok: updated action t
And curling the web action
> curl -v https://openwhisk.ng.bluemix.net/api/v1/web/myspace/default/t.http
< HTTP/1.1 200 OK
< X-Backside-Transport: OK OK
< Connection: Keep-Alive
< Transfer-Encoding: chunked
< Server: nginx/1.11.13
< Date: Tue, 18 Jul 2017 14:00:40 GMT
< Content-Type: text/xml
< Access-Control-Allow-Headers: *
< Access-Control-Allow-Origin: domain
< Access-Control-Allow-Credentials: true
<hi></hi>
Without the annotation:
> wsk action update t t.js -a web-custom-options false --web true
Repeating the curl will show these headers:
< Access-Control-Allow-Origin: *,domain
< Access-Control-Allow-Methods: OPTIONS, GET, DELETE, POST, PUT, HEAD, PATCH
< Access-Control-Allow-Headers: Authorization, Content-Type,*
< Access-Control-Allow-Credentials: true

Use express sendFile for HEAD requests

sendFile is for sending files and it also figures out some interesting headers from the file (like content length). For a HEAD request I would ideally want the exact same headers but just skip the body.
There doesn't seem to be an option for this in the API. Maybe I can override something in the response object to stop it from sending anything?
Here's what I got:
res.sendFile(file, { headers: hdrs, lastModified: false, etag: false })
Has anyone solved this?
As Robert Klep has already written, the sendFile already has the required behavior of sending the headers and not sending the body if the request method is HEAD.
In addition to that, Express already handles HEAD requests for routes that have GET handlers defined. So you don't even need to define any HEAD handler explicitly.
Example:
let app = require('express')();
let file = __filename;
let hdrs = {'X-Custom-Header': '123'};
app.get('/file', (req, res) => {
res.sendFile(file, { headers: hdrs, lastModified: false, etag: false });
});
app.listen(3322, () => console.log('Listening on 3322'));
This sends its own source code on GET /file as can be demonstrated with:
$ curl -v -X GET localhost:3322/file
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 3322 (#0)
> GET /file HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:3322
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< X-Custom-Header: 123
< Accept-Ranges: bytes
< Cache-Control: public, max-age=0
< Content-Type: application/javascript
< Content-Length: 267
< Date: Tue, 11 Apr 2017 10:45:36 GMT
< Connection: keep-alive
<
[...]
The [...] is the body that was not included here.
Without adding any new handler this will also work:
$ curl -v -X HEAD localhost:3322/file
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 3322 (#0)
> HEAD /file HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:3322
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< X-Custom-Header: 123
< Accept-Ranges: bytes
< Cache-Control: public, max-age=0
< Content-Type: application/javascript
< Content-Length: 267
< Date: Tue, 11 Apr 2017 10:46:29 GMT
< Connection: keep-alive
<
This is the same but with no body.
Express uses send to implement sendFile, which already does exactly what you want.

How to properly serve static assets with Hapijs

I'm trying to get Fastly pointed at my heroku instance that is running hapijs and serving static files from /assets/. I have a route setup that looks like this:
// Assets
{
method: "GET",
path: "/assets/{path*}",
config: {
cache: {
privacy: 'public',
expiresIn: 31536000000 // 1 year in milliseconds
}
},
handler: {
directory: { path: './public/assets'}
}
},
Here are the headers sent back on each request:
HTTP/1.1 200 OK
content-type: text/css; charset=utf-8
last-modified: Thu, 22 Jan 2015 07:08:07 GMT
etag: "9c7d48799e5230b7c97ef1978b81ad533c10b950"
set-cookie: csrf=xyz; Path=/
set-cookie: session=xyz; Path=/
cache-control: max-age=31536000, must-revalidate, private
Date: Thu, 22 Jan 2015 07:21:15 GMT
Connection: keep-alive
How do I not set cookies on the responses from this enpoint and why does the cache-control header set must-revalidate and private. Shouldn'it just be public?
I finally solved this issue. I had a few plugins that were setting cookies: yar, hapi-auth-cookie, and crumb. Unfortunately there is not yet any standard way of removing plugins from a particular route.
crumb allows you to add a skip function to the registration options that will disable it.
auth let's you disable it at the route config by setting auth: false.
yar doesn't yet have any mechanism for doing this so I submitted a PR to fix it
You can also remove cookies via Fastly, before the response is considered to be stored in the cache. For this you need to do the following configuration steps:
Content -> Headers:
Name Remove Set-Cookie from /assets
Type: Cache
Action: Delete
Destination: http.Set-Cookie
On that newly generated Header Configuration: Settings -> Cache Conditions ->
New
Name: /assets
Apply If: req.url ~ "^/assets/"
That will remove the Set-Cookie header before it's "seen" by Fastly and will thus make it cachable.

Resources