Finding the final URL that the request will be redirected to in nodeJS? - node.js

I have the following URL:
https://forecast.weather.gov/zipcity.php?inputstring=95014
I would like to figure out which URL it will redirect to. In this example it is:
https://forecast.weather.gov/MapClick.php?CityName=Cupertino&state=CA&site=MTR&lat=37.3042&lon=-122.095
I tried multiple solutions such as:
res.headers.location
and
res.headers.get('location')
But none of them seem to work. I know that the URL redirects because, I could successfully redirect in curl and google-chrome. Here is the code that I am running:
https.get('https://forecast.weather.gov/zipcity.php?inputstring=95014', res => console.log(res.headers.location))
When I was running in curl, I ran the following:
curl -Ls -o /dev/null -w %{url_effective} http://forecast.weather.gov/zipcity.php?inputstring=95014
And got the desired output of:
http://forecast.weather.gov/MapClick.php?CityName=Cupertino&state=CA&site=MTR&lat=37.3042&lon=-122.095
When I run curl:
$ curl -I https://forecast.weather.gov/zipcity.php?inputstring=95014
HTTP/2 302
server: Apache
x-nids-serverid: www1.mo
location: https://forecast.weather.gov/MapClick.php?CityName=Cupertino&state=CA&site=MTR&lat=37.3042&lon=-122.095
x-ua-compatible: IE=Edge
access-control-allow-origin: *
content-type: text/html; charset=UTF-8
content-length: 0
cache-control: max-age=722
expires: Mon, 26 Apr 2021 19:28:03 GMT
date: Mon, 26 Apr 2021 19:16:01 GMT
strict-transport-security: max-age=31536000 ; includeSubDomains ; preload
As you can see, there are multiple headers present here that are not present inside nodejs's results:
{
server: 'AkamaiGHost',
'mime-version': '1.0',
'content-type': 'text/html',
'content-length': '288',
expires: 'Mon, 26 Apr 2021 19:21:50 GMT',
date: 'Mon, 26 Apr 2021 19:21:50 GMT',
connection: 'close',
'strict-transport-security': 'max-age=31536000 ; includeSubDomains ; preload'
}
I would like to stick to the http module.

Related

Curl HEAD request returns a last-modified header but Node https.request(..., {method: HEAD},...) does not

Apparently there's something I don't know about HEAD requests.
Here's the URL: 'https://theweekinchess.com/assets/files/pgn/eurbli22.pgn', which I'll refer to as <URL> below.
If I curl this, I see a last-modified entry in the headers:
curl --head <URL>
HTTP/2 200
last-modified: Sun, 18 Dec 2022 18:07:16 GMT
accept-ranges: bytes
content-length: 1888745
host-header: c2hhcmVkLmJsdWVob3N0LmNvbQ==
content-type: application/x-chess-pgn
date: Wed, 11 Jan 2023 23:09:14 GMT
server: Apache
But if I make a HEAD request in Node using https, That information is missing:
https.request(<URL>, { method: 'HEAD' }, res => {
console.log([<URL>, res.headers])}).end()
This returns:
[
<URL>
{
date: 'Wed, 11 Jan 2023 23:16:15 GMT',
server: 'Apache',
p3p: 'CP="NOI NID ADMa OUR IND UNI COM NAV"',
'cache-control': 'private, must-revalidate',
'set-cookie': [
'evof3sqa=4b412b5913b38669fc928a0cca9870e4; path=/; secure; HttpOnly'
],
upgrade: 'h2,h2c',
connection: 'Upgrade, Keep-Alive',
'host-header': 'c2hhcmVkLmJsdWVob3N0LmNvbQ==',
'keep-alive': 'timeout=5, max=75',
'content-type': 'text/html; charset=UTF-8'
}
]
I tried axios instead of https:
const response = await axios.head('https://theweekinchess.com/assets/files/pgn/eurbli22.pgn');
console.log({response: response.headers})
And that works (incl. the proper MIME type):
date: 'Thu, 12 Jan 2023 20:00:48 GMT',
server: 'Apache',
upgrade: 'h2,h2c',
connection: 'Upgrade, Keep-Alive',
'last-modified': 'Sun, 18 Dec 2022 18:07:16 GMT',
'accept-ranges': 'bytes',
'content-length': '1888745',
'host-header': 'c2hhcmVkLmJsdWVob3N0LmNvbQ==',
'keep-alive': 'timeout=5, max=75',
'content-type': 'application/x-chess-pgn'
I also tried waiting for re2.on('end', console.log(res.headers)), but same output as before.
I'm going to close this issue and post it instead as a 'bug' on Node's site. I'm sure there's something that needs to be changed in how I'm executing the HEAD request.

Getting Error while opening hosted KaiOS app

I have a manifest.webapp hosted at my application root (https://localhost:5001/manifest.webapp), when I open it in KaiOS simulator, I get the following error:
Unable to access the app starting document https://localhost:5001/,
got HTTP code 405
Curl of its response is:
HTTP/1.1 200 OK
Date: Wed, 02 Oct 2019 21:18:41 GMT
Content-Type: application/x-web-app-manifest+json
Content-Length: 6097
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
Last-Modified: Tue, 01 Oct 2019 07:35:10 GMT
Accept-Ranges: bytes
ETag: "1d5782ac10b5cd1"
Set-Cookie: ClientId=; expires=Thu, 01 Jan 1970 00:00:00 GMT; path=/; samesite=lax
Set-Cookie: ClientId=922A46E87C9646C18555E7E7DE84840F; expires=Mon, 02 Oct 2119 21:18:42 GMT; path=/; samesite=lax
Access-Control-Allow-Origin: *
x-frame-options: allow-from https://example.com/
x-web-server-version: 1.0.0.0
x-besku: UNKNOWN
{
"name": "abc",
"short_name": "abc",
"icons": [..],
"scope": "/",
"start_url": "/abc/?start_url=sss",
"display": "standalone",
"background_color": "#ffffff",
"theme_color": "#0078d7"
...
Any help would be appreciated.
Firefox os/kaios makes a HEAD call, before GET, so that should be implemented on your server, else this error will come.

Nginx not Caching my Application

I'm trying to setup the caching of my ExpressApp and its rendered Jade files with Nginx.
When I have a look at the pm2 logs of my application while running the google Pagespeed Insight test, I see that the Images and HTML are still served by the app instead of the Nginx proxy.
Two Questions:
How do I have to change my configuration to have the desired effect?
How do I effectively validate that the Setup/Caching is working?
Thanks!
My current Nginx Configuration:
proxy_cache_path /var/lib/nginx/cache levels=1:2 keys_zone=backcache:8m max_size=50m;
proxy_cache_key "$scheme$request_method$host$request_uri$is_args$args";
proxy_cache_valid 200 302 10m;
proxy_cache_valid 404 1m;
server {
listen 80 default_server;
listen [::]:80 default_server;
location / {
expires 1w;
proxy_cache backcache;
#proxy_cache_bypass $http_cache_control;
add_header X-Proxy-Cache $upstream_cache_status;
proxy_pass http://localhost:3030;
}
}
Headers:
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
* Server certificate: ###
* Server certificate: COMODO RSA Domain Validation Secure Server CA
* Server certificate: COMODO RSA Certification Authority
* Server certificate: AddTrust External CA Root
> GET / HTTP/1.1
> Host: ###
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.10.0 (Ubuntu)
< Date: Thu, 24 Nov 2016 13:15:37 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 66761
< Connection: keep-alive
< Vary: Accept-Encoding
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET,PUT,POST,DELETE
< Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
< ETag: W/"104c9-8+4lo531tUk9k3SJq4sPHg"
< Expires: Thu, 01 Dec 2016 13:15:37 GMT
< Cache-Control: max-age=604800
< X-Proxy-Cache: MISS
Headers of two consecutive Requests to an Image
➜ ~ curl -I **url/img.jpg**
HTTP/1.1 200 OK
Server: nginx/1.10.0 (Ubuntu)
Date: Fri, 25 Nov 2016 03:17:02 GMT
Content-Type: application/octet-stream
Content-Length: 11007
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,PUT,POST,DELETE
Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
Accept-Ranges: bytes
Cache-Control: max-age=604800
Last-Modified: Thu, 17 Nov 2016 16:23:13 GMT
ETag: W/"2aff-158731964ca"
Expires: Fri, 02 Dec 2016 03:17:02 GMT
X-Proxy-Cache: MISS
➜ ~ curl -I **url/img.jpg**
HTTP/1.1 200 OK
Server: nginx/1.10.0 (Ubuntu)
Date: Fri, 25 Nov 2016 03:17:08 GMT
Content-Type: application/octet-stream
Content-Length: 11007
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,PUT,POST,DELETE
Access-Control-Allow-Headers: Origin, x-access-token, X-Requested-With, Content-Type, Accept, un-populated
Accept-Ranges: bytes
Cache-Control: max-age=604800
Last-Modified: Thu, 17 Nov 2016 16:23:13 GMT
ETag: W/"2aff-158731964ca"
Expires: Fri, 02 Dec 2016 03:17:08 GMT
X-Proxy-Cache: MISS
The headers you show as the request headers are most probably not the original ones, because it says "Provisional headers shown".
The actual request headers probably contain the "cache-control" header and your nginx configuration uses that header to bypass the cache, as indicated by the 'X-Proxy-Cache:BYPASS' response header.
If there is no reason to keep it, you should remove the proxy_cache_bypass-directive from your config.

Cannot map custom http header on response using int-http:outbound-gateway

I have an int-http:outbound-gateway to a rest service with a custom http header but this header is not mapped to the SI message headers although is set on the mapped-response-headers config
here's my config:
<int-http:outbound-gateway
request-channel="inChannel" reply-channel="outChannel"
url-expression="http://localhost:8080/rest" mapped-request-headers="HTTP_REQUEST_HEADERS"
http-method-expression="GET" expected-response-type="java.lang.String"
charset="UTF-8"
mapped-response-headers="HTTP_RESPONSE_HEADERS,mySpecificHeader"/>
Here's the raw response from the rest service:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Expires: Fri, 28 Oct 2016 02:05:34 GMT
mySpecificHeader: headerInfo
Content-Type: application/json
Transfer-Encoding: chunked
Date: Fri, 28 Oct 2016 02:05:34 GMT
{"surname":"Constantine","phone":"+33 555 666 777","name":"John","id":12345}

JSF-Login-Page in HTTP-Response despite valid JSESSIONID

I think this a HTTP-related problem.
I want to use my (JAX-RS) RESTeasy Service on a (JEE6) JBoss AS 7 Server from an Android Device. The RESTeasy Service is working fine. I am using on the Client-Side the Restlet-Client. This works too - without Security.
I want to use my JAAS-Formbased Security for the Pattern /rest/* in web.xml. So I have to send a HTTP-POST-Request with the Form-Data (j_username and j_password) to /foo/j_security_check.
I get the JSESSIONID from the first Response by the Server:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=uKUqlkUWdhX2l-FihiWyeSJr.undefined; Path=/foo
Pragma: No-cache
Cache-Control: no-cache
Expires: Thu, 01 Jan 1970 01:00:00 CET
X-Powered-By: JSF/2.0
Content-Type: text/html;charset=utf-8
Content-Length: 1028
Date: Wed, 15 Aug 2012 11:42:59 GMT
For this anonymous session I am authenticating ...
Header:
POST /foo/j_security_check HTTP/1.1
Date: Wed, 15 Aug 2012 11:42:58 GMT
Accept: text/html
Host: 172.24.47.5:8080
User-Agent: Restlet-Framework/2.0.14
Cookie: JSESSIONID=uKUqlkUWdhX2l-FihiWyeSJr.undefined
Content-Length: 62
Content-Language: *
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content:
j_username=Bob&j_password=a
... and it works: JBoss-Security-TRACE:
2012-08-15 13:22:26,829 TRACE
[org.jboss.security.auth.spi.DatabaseServerLoginModule]
(http-0.0.0.0-0.0.0.0-8080-4) User 'Bob' authenticated, loginOk=true
Now the Problem: In the following request I want to GET the REST-URL (using the Cookie JSESSIONID):
GET /foo/rest/sync/products HTTP/1.1
Date: Wed, 15 Aug 2012 11:42:59 GMT
Accept: application/json
Host: 172.24.47.5:8080
User-Agent: Restlet-Framework/2.0.14
Cookie: JSESSIONID=uKUqlkUWdhX2l-FihiWyeSJr.undefined
Content-Length: 0
But instead of returning the Response with JSON Content, the server is returning the JSF-Login-Page, because it want's me to authenticate again(?):
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Pragma: No-cache
Cache-Control: no-cache
Expires: Thu, 01 Jan 1970 01:00:00 CET
X-Powered-By: JSF/2.0
Content-Type: text/html;charset=utf-8
Content-Length: 936
Date: Wed, 15 Aug 2012 11:42:59 GMT
<?xml version="1.0" encoding="utf-8"?> ... ... ... </html>
If I login with the Browser and then open the REST-URL it works fine. This is the GET-Request by the Browser:
GET http://localhost:8080/foo/rest/sync/products HTTP/1.1
Host: localhost:8080
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.5 (KHTML, like Gecko)
Chrome/19.0.1084.56 Safari/536.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: JSESSIONID=royq26yLd7REOz2otiZdTl6j.undefined
Anyone has an idea? I think the problem lays in the last request (GET /foo/rest/sync/products), because in the Browser it works fine.
Thanks

Resources