ExpressJs retry query after long time processing - node.js

We are having a weird problem with our Nodejs-Expressjs server. We are trying to perform a heavy query that takes 5-10 minutes to finish. The query works well, but for some reason after 2-4 minutes the server retry the query again.
We have traceability logging implanted and we see the query is performed twice with the same requested.
Does anyone have an idea of the possible reasons for this behavior?
,Request Morgan --> ,77e53abf-272c-420d-9e82-**d59736a2f169,POST,undefined,/admin/cars/feed,undefined ms,# Thu, 29 Apr 2021 07:02:25 GMT,127.0.0.1,from http://localhost:8085/,Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Mobile Safari/537.36, Size= ,undefined,bytes,**
undefined
[LOGGER] 21-04-29 09:04:30 info : Receiving JWT payload {"sub":{"userId":"fc32df37-ee2b-44e3-a864-8b73ea31d1aa","roleId":2},"tokenId":"RReUCy7UD","type":0,"iat":1619675641}
77e53abf-272c-420d-9e82-d59736a2f169
Your token is VALID
[LOGGER] 21-04-29 09:04:31 info : User data found in cache
77e53abf-272c-420d-9e82-d59736a2f169
[LOGGER] 21-04-29 09:04:32 debug : User "baaldev" authenticated
77e53abf-272c-420d-9e82-d59736a2f169
[LOGGER] 21-04-29 09:04:32 info : Validating POST:/cars/feed
undefined
[LOGGER] 21-04-29 09:04:32 info : /admin/cars/feed request with data
filters=[{"carOwnerType":{"value":[{"value":"hideScrapped","label":"SEARCHCAR.FROM_SCRAPPED"}]}}]
name=t
platform=trovit
undefined
[LOGGER] 21-04-29 09:04:33 info : User is authenticated to request
undefined
[LOGGER] 21-04-29 09:04:62 debug : Getting cars
77e53abf-272c-420d-9e82-d59736a2f169
[LOGGER] 21-04-29 09:04:87 info : Executing (default): SELECT count(`CarsForSelling`.`carId`) AS `count` FROM `CarsForSelling` AS `CarsForSelling` LEFT OUTER JOIN `UserInformation` AS `OwnedCars` ON `CarsForSelling`.`userId` = `OwnedCars`.`userId` LEFT OUTER JOIN `BusinessInformation` AS `OwnedCars->BusinessInformation` ON `OwnedCars`.`userId` = `OwnedCars->BusinessInformation`.`userId` WHERE `CarsForSelling`.`status` > 0 AND `CarsForSelling`.`lockStatus` = '0' AND `CarsForSelling`.`validUntil` >= '2021-04-29 07:02:32';
undefined
[LOGGER] 21-04-29 09:04:29 http : HTTP log
77e53abf-272c-420d-9e82-d59736a2f169
[LOGGER] 21-04-29 09:04:29 info :
**,Request Morgan --> ,77e53abf-272c-420d-9e82-d59736a2f169,POST,undefined,/admin/cars/feed,undefined ms,# Thu, 29 Apr 2021 07:04:26 GMT,127.0.0.1,from http://localhost:8085/,Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Mobile Safari/537.36, Size= ,undefined,bytes,**
undefined
[LOGGER] 21-04-29 09:04:78 info : Executing (default): SELECT `CarsForSelling`.*, `CarImages`.`imageId` AS `CarImages.imageId`, `CarImages`.`carId` AS `CarImages.carId`, `CarImages`.`imageName` AS `CarImages.imageName`, `CarImages`.`imageUrl` AS `CarImages.imageUrl`, `CarImages`.`order` AS `CarImages.order`, `CarImages`.`imageSourceUrl` AS `CarImages.imageSourceUrl`, `CarImages`.`createdAt` AS


It is why we have status code 202
function endPoint (req, res) {
// code
//Do not put a `return`
res.sendStatus(202);
// exec query
const result = runQuery()
return;
}
The HyperText Transfer Protocol (HTTP) 202 Accepted response status
code indicates that the request has been accepted for processing, but
the processing has not been completed; in fact, processing may not
have started yet. The request might or might not eventually be acted
upon, as it might be disallowed when processing actually takes place.
202 is non-committal, meaning that there is no way for the HTTP to
later send an asynchronous response indicating the outcome of
processing the request. It is intended for cases where another process
or server handles the request, or for batch processing.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/202

Related

Stream audio from S3 to client through REST API

Description
I am building a music streaming SPA using NEXT.js.
Audio files are stored on AWS S3.
The goal is to stream audio file from S3 to client through REST
so that authentication is possible, and to "hide" the AWS endpoints.
Issue
When streaming data down to client through REST endpoint the audio glitches and loads only ~15seconds of the audio file being played.
I tested this behaviour on separate project with manually creating the read stream and providing options to it :
fs.createReadStream("path", {start: startByte, end: endByte})
and it works just fine.
Although the createReadStream from s3 (i believe im using v2) does not accept any options. So i am unable to fix this glitching this way.
I thought about many solutions one of which involved manually converting the incoming buffer from S3 to streamable data, but this will lead to data being processed in server's RAM i believe, and i dont want this behaviour even though audio files are usually quite "small".
I also thought about creating a presigned url to the file and then redirecting in the worst case scenario.
Question
I will provide source code below. I believe my audio loops on first ~15 seconds due to readstream lacking start and end positions.
How do i fix given behaviour and stream data corectly from s3 to server to client without saving whole files in servers RAM?
Code
Part of the utility function for data streaming:
const downloadParams = {
Key,
Bucket: bucketName,
};
const fileStream = s3.getObject(downloadParams).createReadStream();
fileStream is returned from this function and accessed in API endpoint like so:
const CHUNK_SIZE = 10 ** 3 * 500; // ~500KB
const startByte = Number(range.replace(/\D/g, ""));
const endByte = Math.min(
startByte + CHUNK_SIZE,
attr.ObjectSize - 1
);
const chunk = endByte - startByte + 1;
const headers = {
"Content-Range": `bytes ${startByte}-${endByte}/${attr.ObjectSize}`,
"Accept-Ranges": "bytes",
"Content-Length": chunk,
"Content-Type": "audio/*",
};
res.writeHead(206, headers);
fileStream.pipe(res);
Here is audio receiver on the client:
"use client";
const Audio = () => {
return (
<audio src="http://localhost:3000/api/stream/FILE_KEY_HERE" controls></audio>
);
};
export default Audio;
here is how request headers look like:
Accept: */*
Accept-Encoding: identity;q=1, *;q=0
Accept-Language: en,ru;q=0.9,sv-SE;q=0.8,sv;q=0.7,en-US;q=0.6
Connection: keep-alive
Cookie:
Host: localhost:3000
Range: bytes=65536-
Referer: http://localhost:3000/
sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
sec-ch-ua-mobile: ?1
sec-ch-ua-platform: "Android"
Sec-Fetch-Dest: video
Sec-Fetch-Mode: no-cors
Sec-Fetch-Site: same-origin
sec-gpc: 1
User-Agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36
second sheader differs only by Range: bytes=65536-
first request:
Request URL: http://localhost:3000/api/stream/track/4
Request Method: GET
Status Code: 206 Partial Content
Remote Address: [::1]:3000
Referrer Policy: strict-origin-when-cross-origin
Response Headers:
Accept-Ranges: bytes
Connection: keep-alive
Content-Length: 500001
Content-Range: bytes 65536-565536/3523394
Content-Type: audio/*
Date: Wed, 25 Jan 2023 21:35:51 GMT
Keep-Alive: timeout=5
PS
I did check my network tab and headers contain thruthful information about objects that are being streamed. Requests seem to download full file size (3.2mb of 3.2mb for example), but the audio still loops on first 15 seconds. Even if i manipulate the duration bar manually.
Haven't found any information like this here so thought this would be helpful to someone in the future
Tried
On top of things mentioned I tried creating new streams and piping them, tried using stream events on createReadStream(), read poorly written aws docs. But due to lack of info it is less time consuming to ask someone than trying fixing same issue for 4 days straight.

The issue is that the first X bytes were read from the source MP3, regardless of if the client requested a later 'range'.
The quick solution was to just tell the GetObject function to seek to the same bytes the request states in the Range header, since S3 itself also supports range requests.

TOO_MANY_REDIRECTS error when iFraming Kibana dashboard using cookies

I'm attempting to embed password protected Kibana dashboards inside an iFrame to my Node powered Express application, based on information in this thread. Kibana is protected using X-Pack and requires users to login in order to see their visualised data.
This currently requires the user to log in twice, once to login into the application and again to access Kibana dashboards, which is not the goal.
Following information in this thread, I implemented some code that makes a pre-flight POST request to https://elk-stack.my.domain:5601/api/security/v1/login to obtain a cookie 🍪
This client side request...
function preFlightKibanaAuth () {
...
$.ajax({
type: 'POST',
url: '/kibana-auth',
datatype: 'json',
success: function (response) {
if (response && response.authenticated) {
$('iframe#kibana-dashboard').prop('src', 'https://elk-stack.my.domain:5601/s/spacename/app/kibana#/dashboards?_g=()')
}
},
error: function (err) {
console.log(err)
}
})
}
Is routed to this route...
router
.route('/kibana-auth')
.post((req, res, next) => {
...
if (authorised) {
...
authenticateKibana(req)
.then(cookie => {
if (cookie && cookie.name && cookie.value) {
res.set('Set-Cookie', `${cookie.name}=${cookie.value}; Domain=my.domain; Path=/; Secure; HttpOnly`)
res.send({ 'authenticated': true })
} else {
res.send({ 'authenticated': false })
}
})
.catch((err) => {
logger.error(err)
res.send({ 'authenticated': false })
})
}
...
})
Which makes it's way to this function, where the cookie is actually obtained and parsed...
authenticateKibana () {
return new Promise((resolve, reject) => {
...
request({
method: 'POST',
uri: `https://elk-stack.my.domain:5601/api/security/v1/login`,
headers: {
'kibana-version': '6.5.4',
'kibana-xsrf': 'xsrftoken',
},
type: 'JSON',
form: {
password: 'validPassword',
username: 'validUsername'
}
}, function (error, res, body) {
if (!error) {
let cookies = cookieParser.parse(res)
cookies.forEach(function (cookie) {
if (cookie.name.startsWith('kibana')) {
// Got Kibana Cookie
return resolve(cookie)
}
})
}
...
})
})
}
This works great and I can successfully authenticate with Kibana, obtain the cookie and set in the clients browser (see below screenshot).
The issue I'm seeing is when the src of the iFrame is updated in the success callback of the preFlightKibanaAuth() request. I can see the authenticated Kibana dashboard load in briefly (so the cookie is allowing the client to view their authenticated dashboards), however, I then see multiple GET requests to /login?next=%2Fs%2Fspacename%2Fapp%2Fkibana that results in a TOO_MANY_REDIRECTS error.
I've found the below comment in the GitHub issues page, which I think maybe the issue I'm having in some way because I'm seeing this in the logs (see bottom): "message":"Found 2 auth sessions when we were only expecting 1.". I just can't figure it out!
Usually what causes this is having multiple cookies for the same
"domain" and "name", but with different values for "path". If you open
the developer tools in Chrome, then click on the Application tab, then
expand the Cookies section, and click on the domain, do you have
multiple cookies with the name "sid"? If so, you can fix this issue by
clearing all of them.
I changed the cookie name from "sid" to "kibana" but don't have two of them visible in Applications tab, just the one I set following the call to /kibana-auth.
The iFrame then loads in the https://elk-stack.my.domain:5601/s/spacename/app/kibana#/dashboards?_g=() and the issue arises. Clearing my cookies just resulted in fetching and setting another one (if we don't already have one), which is what is required, so this didn't solve the problem.
When I send the Set-Cookie header back to the client, I am setting the Domain to the main domain: my.domain, which ends up as .my.domain. The Kibana instance is on a subdomain: elk-stack.my.domain and if I login to the Kibana front end, I can see that the Domain of the cookie it returns is set to elk-stack.my.domain. I'm not sure that should matter though.
Can anyone please shed any light on this or point me in the direction?
Thanks in advance
Here's a glimpse at the logging info from /var/log/kibana/kibana.stdout when a request is made. There's a bit of junk in there still but you can still see what's happening.
{"type":"log","#timestamp":"2019-02-12T19:47:44Z","tags":["debug","security","basic"],"pid":7857,"message":"Trying to authenticate user request to /api/security/v1/login."}
{"type":"log","#timestamp":"2019-02-12T19:47:44Z","tags":["debug","security","basic"],"pid":7857,"message":"Trying to authenticate via header."}
{"type":"log","#timestamp":"2019-02-12T19:47:44Z","tags":["debug","security","basic"],"pid":7857,"message":"Request has been authenticated via header."}
{"type":"response","#timestamp":"2019-02-12T19:47:44Z","tags":[],"pid":7857,"method":"post","statusCode":204,"req":{"url":"/api/security/v1/login","method":"post","headers":{"kibana-version":"6.5.4","kbn-xsrf":"6.5.4","host":"10.30.10.30:5601","content-type":"application/
x-www-form-urlencoded","content-length":"35","connection":"close"},"remoteAddress":"192.168.56.102","userAgent":"192.168.56.102"},"res":{"statusCode":204,"responseTime":109,"contentLength":9},"message":"POST /api/security/v1/login 204 109ms - 9.0B"}
{"type":"log","#timestamp":"2019-02-12T19:47:44Z","tags":["debug","legacy","proxy"],"pid":7857,"message":"Event is being forwarded: connection"}
{"type":"log","#timestamp":"2019-02-12T19:47:44Z","tags":["trace","legacy","service"],"pid":7857,"message":"Request will be handled by proxy GET:/s/spacename/app/kibana."}
{"type":"log","#timestamp":"2019-02-12T19:47:44Z","tags":["warning","security","auth","session"],"pid":7857,"message":"Found 2 auth sessions when we were only expecting 1."}
{"type":"log","#timestamp":"2019-02-12T19:47:44Z","tags":["debug","security","basic"],"pid":7857,"message":"Trying to authenticate user request to /app/kibana."}
{"type":"log","#timestamp":"2019-02-12T19:47:44Z","tags":["debug","security","basic"],"pid":7857,"message":"Trying to authenticate via header."}
{"type":"log","#timestamp":"2019-02-12T19:47:44Z","tags":["debug","security","basic"],"pid":7857,"message":"Authorization header is not presented."}
{"type":"response","#timestamp":"2019-02-12T19:47:44Z","tags":[],"pid":7857,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"host":"elk-stack.my.domain:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8","referer":"https://local.local.my.domain/fortigate/reporting/dashboard","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en;q=0.9,en-US;q=0.8,la;q=0.7,fr;q=0.6"},"remoteAddress":"192.168.56.102","userAgent":"192.168.56.102","referer":"https://local.local.my.domain/fortigate/reporting/dashboard"},"res":{"statusCode":302,"responseTime":3,"contentLength":9},"message":"GET /app/kibana 302 3ms - 9.0B"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","legacy","proxy"],"pid":7857,"message":"Event is being forwarded: connection"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","legacy","proxy"],"pid":7857,"message":"\"getConnections\" has been called."}
{"type":"ops","#timestamp":"2019-02-12T19:47:45Z","tags":[],"pid":7857,"os":{"load":[0.2568359375,0.31640625,0.3173828125],"mem":{"total":33567580160,"free":346796032},"uptime":1585351},"proc":{"uptime":33636.577,"mem":{"rss":322772992,"heapTotal":225566720,"heapUsed":184707176,"external":2052484},"delay":6.417333126068115},"load":{"requests":{"5601":{"total":2,"disconnects":0,"statusCodes":{"204":1,"302":1}}},"concurrents":{"5601":1},"responseTimes":{"5601":{"avg":56,"max":109}},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 176.2MB uptime: 9:20:37 load: [0.26 0.32 0.32] delay: 6.417"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":7857,"message":"Received Kibana Ops event data"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","monitoring-ui","kibana-monitoring"],"pid":7857,"message":"Received Kibana Ops event data"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["trace","legacy","service"],"pid":7857,"message":"Request will be handled by proxy GET:/login?next=%2Fs%2Fspacename%2Fapp%2Fkibana."}
{"type":"response","#timestamp":"2019-02-12T19:47:45Z","tags":[],"pid":7857,"method":"get","statusCode":302,"req":{"url":"/login?next=%2Fs%2Fspacename%2Fapp%2Fkibana","method":"get","headers":{"host":"elk-stack.my.domain:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8","referer":"https://local.local.my.domain/fortigate/reporting/dashboard","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en;q=0.9,en-US;q=0.8,la;q=0.7,fr;q=0.6"},"remoteAddress":"192.168.56.102","userAgent":"192.168.56.102","referer":"https://local.local.my.domain/fortigate/reporting/dashboard"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"GET /login?next=%2Fs%2Fspacename%2Fapp%2Fkibana 302 2ms - 9.0B"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","legacy","proxy"],"pid":7857,"message":"Event is being forwarded: connection"}
The below then repeats over and over...
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["trace","legacy","service"],"pid":7857,"message":"Request will be handled by proxy GET:/s/spacename/app/kibana."}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["warning","security","auth","session"],"pid":7857,"message":"Found 2 auth sessions when we were only expecting 1."}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","security","basic"],"pid":7857,"message":"Trying to authenticate user request to /app/kibana."}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","security","basic"],"pid":7857,"message":"Trying to authenticate via header."}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","security","basic"],"pid":7857,"message":"Authorization header is not presented."}
{"type":"response","#timestamp":"2019-02-12T19:47:45Z","tags":[],"pid":7857,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"host":"elk-stack.my.domain:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8","referer":"https://local.local.my.domain/fortigate/reporting/dashboard","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en;q=0.9,en-US;q=0.8,la;q=0.7,fr;q=0.6"},"remoteAddress":"192.168.56.102","userAgent":"192.168.56.102","referer":"https://local.local.my.domain/fortigate/reporting/dashboard"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"GET /app/kibana 302 2ms - 9.0B"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","legacy","proxy"],"pid":7857,"message":"Event is being forwarded: connection"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["trace","legacy","service"],"pid":7857,"message":"Request will be handled by proxy GET:/login?next=%2Fs%2Fspacename%2Fapp%2Fkibana."}
{"type":"response","#timestamp":"2019-02-12T19:47:45Z","tags":[],"pid":7857,"method":"get","statusCode":302,"req":{"url":"/login?next=%2Fs%2Fspacename%2Fapp%2Fkibana","method":"get","headers":{"host":"elk-stack.my.domain:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8","referer":"https://local.local.my.domain/fortigate/reporting/dashboard","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en;q=0.9,en-US;q=0.8,la;q=0.7,fr;q=0.6"},"remoteAddress":"192.168.56.102","userAgent":"192.168.56.102","referer":"https://local.local.my.domain/fortigate/reporting/dashboard"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"GET /login?next=%2Fs%2Fspacename%2Fapp%2Fkibana 302 2ms - 9.0B"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","legacy","proxy"],"pid":7857,"message":"Event is being forwarded: connection"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["plugin","debug"],"pid":7857,"message":"Checking Elasticsearch version"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["trace","legacy","service"],"pid":7857,"message":"Request will be handled by proxy GET:/s/spacename/app/kibana."}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["warning","security","auth","session"],"pid":7857,"message":"Found 2 auth sessions when we were only expecting 1."}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","security","basic"],"pid":7857,"message":"Trying to authenticate user request to /app/kibana."}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","security","basic"],"pid":7857,"message":"Trying to authenticate via header."}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","security","basic"],"pid":7857,"message":"Authorization header is not presented."}
{"type":"response","#timestamp":"2019-02-12T19:47:45Z","tags":[],"pid":7857,"method":"get","statusCode":302,"req":{"url":"/app/kibana","method":"get","headers":{"host":"elk-stack.my.domain:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8","referer":"https://local.local.my.domain/fortigate/reporting/dashboard","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en;q=0.9,en-US;q=0.8,la;q=0.7,fr;q=0.6"},"remoteAddress":"192.168.56.102","userAgent":"192.168.56.102","referer":"https://local.local.my.domain/fortigate/reporting/dashboard"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"GET /app/kibana 302 2ms - 9.0B"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","legacy","proxy"],"pid":7857,"message":"Event is being forwarded: connection"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["trace","legacy","service"],"pid":7857,"message":"Request will be handled by proxy GET:/login?next=%2Fs%2Fspacename%2Fapp%2Fkibana."}
{"type":"response","#timestamp":"2019-02-12T19:47:45Z","tags":[],"pid":7857,"method":"get","statusCode":302,"req":{"url":"/login?next=%2Fs%2Fspacename%2Fapp%2Fkibana","method":"get","headers":{"host":"elk-stack.my.domain:5601","connection":"keep-alive","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8","referer":"https://local.local.my.domain/fortigate/reporting/dashboard","accept-encoding":"gzip, deflate, br","accept-language":"en-GB,en;q=0.9,en-US;q=0.8,la;q=0.7,fr;q=0.6"},"remoteAddress":"192.168.56.102","userAgent":"192.168.56.102","referer":"https://local.local.my.domain/fortigate/reporting/dashboard"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"GET /login?next=%2Fs%2Fspacename%2Fapp%2Fkibana 302 2ms - 9.0B"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["debug","legacy","proxy"],"pid":7857,"message":"Event is being forwarded: connection"}
{"type":"log","#timestamp":"2019-02-12T19:47:45Z","tags":["plugin","debug"],"pid":7857,"message":"Checking Elasticsearch version"}
Kibana Version: 6.5.4
Elasticsearch: 6.5.4

At first, I thought this all turned out to be a mismatch in the Cookie attributes, alas, it wasn't!
Received some info from the Elastic team...
The cookie which Kibana replies with generally sets the httpOnly flag,
and the secure flag (when hosted over https), in addition to the
domain. If any of the settings differ for the cookie which you're
trying to force Kibana to use, you'll see 2 cookies being submitted
and behaviour similar to what you're seeing.
Thought I was setting the cookie with different attributes, but wasn't... ended up using a plugin to get this off the ground: https://readonlyrest.com/

using nodejs, express and basic auth how do i get username and password

I am using nodejs, express
A third party that will be calling my api will be using basic auth in the format of http://{username}:{password}#yourdomain.com/
I have tried
var auth = require('basic-auth')
app.use(function(req, res, next) {
var credentials = auth(req)
if (req.headers.authorization) {
console.log("found headers");
}
}
But this only works if the basic auth is passed in the header.
I can not seem to get the username and password from the URL(which is the only way this external party can call my api)
I then tried as suggested use url
here is what i am seeing now when i do a POST to
http://myusername:mypassword#localhost:4050/api/callback
var express = require('express');
var http = require('http');
var url = require('url');
var app = express();
app.use(function(req, res, next) {
console.log("req.protocol=",req.protocol);
console.log("req.get('host')=",req.get('host'));
console.log("req.originalUrl=",req.originalUrl);
}
http.createServer(app).listen(config.port, function () {
console.log("HTTP BandWidth listening on port " + config.port);
});
My console looks like
req.protocol= http
req.get('host')= localhost:4050
req.originalUrl=/api/callback
if i dump the entire req obj i do not see myusername or mypassword
i must be missing something obvious
Thanks
Randy

You should be able to use the built-in url package in Node.
https://nodejs.org/dist/latest-v6.x/docs/api/url.html#url_urlobject_auth
const url = require('url');
app.use(function(req, res, next) {
const urlObj = url.parse(req.protocol + '://' + req.get('host') + req.originalUrl);
console.log('Auth info: ' + urlObj.auth);
}
Hope this helps!
EDIT: Well, I take that back. It looks like the use of username and password in the URI has been deprecated, and browsers may just be ignoring that information. See RFC 3986:
3.2.1. User Information
The userinfo subcomponent may consist of a user name and,
optionally, scheme-specific information about how to gain
authorization to access the resource. The user information, if
present, is followed by a commercial at-sign ("#") that delimits it
from the host.
userinfo = *( unreserved / pct-encoded / sub-delims / ":" )
Use of the format "user:password" in the userinfo field is
deprecated. Applications should not render as clear text any data
after the first colon (":") character found within a userinfo
subcomponent unless the data after the colon is the empty string
(indicating no password). Applications may choose to ignore or
reject such data when it is received as part of a reference and
should reject the storage of such data in unencrypted form. The
passing of authentication information in clear text has proven to be
a security risk in almost every case where it has been used.
And...
7.5. Sensitive Information
URI producers should not provide a URI that contains a username or
password that is intended to be secret. URIs are frequently
displayed by browsers, stored in clear text bookmarks, and logged by
user agent history and intermediary applications (proxies). A
password appearing within the userinfo component is deprecated and
should be considered an error (or simply ignored) except in those
rare cases where the 'password' parameter is intended to be public.
I added the bold and italics...
I tried this using the developer tools in Chrome and got the following:
General
Request URL:http://foobar:password#localhost:8888/test
Request Method:GET
Status Code:200 OK
Remote Address:127.0.0.1:8888
Response Headers
Connection:keep-alive
Content-Length:19
Content-Type:application/json; charset=utf-8
Date:Thu, 08 Dec 2016 03:52:35 GMT
ETag:W/"13-uNGID+rxNJ6jDZKj/wrpcA"
Request Headers
GET /test HTTP/1.1
Host: localhost:8888
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
So, it doesn't even look like the username and password info is being passed along by Chrome. Unfortunately, I think you're out of luck if you're trying to use this schema. You may have to set the authorization headers, set your own custom headers (which is what I have done in the past), or pass your credentials in the query string.

AWS 504 gateway timeout - early response from express middleware

I see a 504 gateway timeout when running my service on AWS. Per my service logs, looks like the middleware returned early with a blank response, but the request to the 3rdPartyService did succeed. I am unable to reproduce this issue locally.
Here are the details:
MyService <---> 3rdPartyService
3rdPartyService takes over 2 min to respond, but does succeed ultimately.
MyService is using express on nodeJS, and is deployed using a docker container onto AWS. MyService is set to timeout well after the AWS ELB timeout which is set to 5 min.
Per the log snippet below, see line #16, where the blank response with status 200 is returned. And the 3rdPartyService request succeeding in the very last line.
MyService log snippet:
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"status":"PENDING","msg":"","time":"2016-09-19T21:10:09.973Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"msg":"retrying function getLocation in 5000 ms : attempts: 1","time":"2016-09-19T21:10:09.975Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"serviceRequestId":"da7fae85-f624-4338-adbe-4fc8674cd1","msg":"","time":"2016-09-19T21:10:46.037Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"status":"PENDING","msg":"","time":"2016-09-19T21:10:48.354Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"msg":"retrying function getLocation in 5000 ms : attempts: 2","time":"2016-09-19T21:10:48.354Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"serviceRequestId":"da7fae85-f624-4338-adbe-4fc8674cd1","msg":"","time":"2016-09-19T21:10:56.424Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"status":"PENDING","msg":"","time":"2016-09-19T21:11:00.365Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"msg":"retrying function getLocation in 5000 ms : attempts: 3","time":"2016-09-19T21:11:00.365Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"serviceRequestId":"da7fae85-f624-4338-adbe-4fc8674cd1","msg":"","time":"2016-09-19T21:11:07.049Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"status":"PENDING","msg":"","time":"2016-09-19T21:11:08.508Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"msg":"retrying function getLocation in 5000 ms : attempts: 4","time":"2016-09-19T21:11:08.509Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"serviceRequestId":"da7fae85-f624-4338-adbe-4fc8674cd1","msg":"","time":"2016-09-19T21:11:17.422Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"status":"PENDING","msg":"","time":"2016-09-19T21:11:19.128Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"msg":"retrying function getLocation in 5000 ms : attempts: 5","time":"2016-09-19T21:11:19.129Z","v":0}
{"name":"myservice-api","hostname":"4658d629c774","pid":19,"level":30,"method":"GET","url":"/path/b7446b06-7ac6-40c1-94cd-08fbade4387f/location","status":200,"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36","version":"1.0","ipAddress":"XX.XX.XX.XX","time":129711.250805,"msg":"","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"serviceRequestId":"da7fae85-f624-4338-adbe-4fc8674cd1","msg":"","time":"2016-09-19T21:11:26.294Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"status":"PENDING","msg":"","time":"2016-09-19T21:11:27.995Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"msg":"retrying function getLocation in 5000 ms : attempts: 6","time":"2016-09-19T21:11:27.996Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"serviceRequestId":"da7fae85-f624-4338-adbe-4fc8674cd1","msg":"","time":"2016-09-19T21:11:36.641Z","v":0}
{"name":"adapters/demo_adapter","hostname":"4658d629c774","pid":19,"level":30,"coord":{"lat":37.39948777777778,"lon":-102.045195},"status":"SUCCESS","msg":"","time":"2016-09-19T21:11:39.416Z","v":0}

groovy,Jsoup, get status code

I hope someone can help me with this as I have been searching for it but didn't find anything working.
I am connecting to a number of urls from a list and everything works fine but then I started getting a 404 error on some hence now I want to catch the error so that the program doesn't terminate and keeps going through the list urls.
This is the error I got
org.jsoup.HttpStatusException: HTTP error fetching URL. Status=404, URL=http:nameofthesite
I am using Jsoup.connect and the error is caused in this line of code
Document doc= Jsoup.connect(countryUrl[i2]).timeout(10*1000)
.userAgent("Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5")
.get();
How can I change the code so that I can get the status code.
I've tried Connection.response (something I found on this site as a solution for this sort of problem) but I was getting casting error
Connection.response response= Jsoup.connect(countryUrl[i2]).timeout(10*1000)
.userAgent("Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5")
.execute();
int statusCode = response.statusCode();
if(statusCode==200){
but I get the following error
groovy.lang.MissingMethodException: No signature of method: static org.jsoup.Connection.response() is applicable for argument types: (org.jsoup.helper.HttpConnection$Response) values: [org.jsoup.helper.HttpConnection$Response#c7325ae]
Possible solutions: respondsTo(java.lang.String), respondsTo(java.lang.String, [Ljava.lang.Object;)
any help will be appreciated, thanks.

There's a typo in your code:
Connection.response response = Jsoup.connect(...) ...
// ^
// |
Response is a static class (interface to be correct) of Connection, so just change your code:
Connection.Response response = Jsoup.connect(...) ...

Resources