Heroku and Chaining Nested HTTP Requests (Alamofire/NodeJS) - node.js

I am attempting to nest two get requests inside of each other
Alamofire.request(.GET, "serverAddress")
.responseJSON { response in
if let JSON = response.result.value {
//populate parameters with the expected args
var parameters:[String:[String]] = [String:[String]]()
//Request2
self.queryFriendsList(parameters)
}
Here is the nested request within a different method...
func queryFriendsList(parameters:[String:[String]]) {
Alamofire.request(.GET, "serverAddress", parameters : parameters, encoding: .JSON)
.responseJSON {
response in
if let res = response.result.value {
print(res)
}
}
}
When I check my logs on Heroku I see:
heroku[router]: sock=client at=warning code=H27 desc="Client Request Interrupted" method=GET path="/query_friends_list/"
The log associated with the app reports:
Error: request aborted
at IncomingMessage.onAborted (/app/node_modules/raw- body/index.js:269:10)
at emitNone (events.js:67:13)
at IncomingMessage.emit (events.js:166:7)
at abortIncoming (_http_server.js:281:11)
at emitOne (events.js:82:20)
at Socket.serverSocketCloseListener (_http_server.js:294:5)
at Socket.emit (events.js:169:7)
The second request is dependent upon the first succeeding.
I am at a loss as to how to debug this one. I am using the free version of Heroku, so I wasn't able to determine from the docs if multiple requests would be an issue ?

The issue with this bug had to do with me not understanding where to look for the error...
sock=client
Denoted that the issue was associated with the client side. Initially I had specified a get request for my http request to heroku. I specified parameters to be encoded with JSON but I realized I cannot send a payload encoded with JSON on a get request.
This was really the root issue at hand. After switching my route to a post request and updating the calls in my swift project, I was able to make the proper query against my DB!

Related

GET request using node/request throwing unexpected "read ECONNRESET"

I am getting strange behavior using the request library in node when hitting certain sites.
for a quick context (that has nothing to do with the technical issue): my goal on this project was to read the NASDAQ dividend history for a number of stocks. The histories can be quite long and I'm certainly not going to sit here and read 100 of them manually. An example of one of these pages: https://www.nasdaq.com/symbol/msft/dividend-history
On to the technical troubles... I am using node v11.10.0 with the request module to call the page - the simplest possible thing. I have used this library to do other similar tasks and have never had an issue. Here was my very simple script:
var request = require("request");
var options = { method: 'GET',
url: 'https://www.nasdaq.com/symbol/msft/dividend-history',
headers:
{ 'Postman-Token': 'e4e9749e-d73e-44b9-994f-9d0522654fb0',
'cache-control': 'no-cache' } };
request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(body);
});
Save file as index.js, and run
node index.js
and that should get the page... except I'm getting an error. The error is:
Error: Error: read ECONNRESET
at Request._callback (<localpath>\index.js:8:24)
at self.callback (<localpath>\node_modules\request\request.js:185:22)
at Request.emit (events.js:197:13)
at Request.onRequestError (<localpath>\node_modules\request\request.js:881:8)
at ClientRequest.emit (events.js:197:13)
at TLSSocket.socketErrorListener (_http_client.js:397:9)
at TLSSocket.emit (events.js:197:13)
at emitErrorNT (internal/streams/destroy.js:82:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
at processTicksAndRejections (internal/process/next_tick.js:76:17)
So, thinking that I maybe had something wrong, I tried the same request in postman to validate. But Postman had no trouble doing a get request against the same URL above. I thought perhaps there was a certificate issue with the https stuff, but I toggled certificate validation on and off in postman, and it worked regardless of the setting.
Additional note here - my "very simple script" is actually what I pulled from the Postman "code" tab for a Node/Request code block, so I am really baffled that Postman works but Postman's own node codeblock does not.
At this point I don't even care about what I was trying to do originally... I'm just trying to figure out why I am getting different behavior on two seemingly equivalent ways to make a GET request.
Does anyone have insight on what postman has that node is missing in this case?

Node.js Request() failing on pages of one specific domain

My customer is all of a sudden experiencing problems with a HTML scraper job made with Node.js. I have circled in on the cause, and found that it's located in the Request module. That made me write a small test application, which solely gets the HTML of the given URL via the Request module. Like this:
var request = require('request');
request('https://www.politi.dk/da/ompolitiet/jobipolitiet/ledige_stillinger/ledigestillinger', function(err, res, body){
if(err){
console.log(err);
} else {
console.log('statusCode:', res.statusCode);
console.log('statusMessage:', res.statusMessage);
}
});
The above example does not work though, as I am getting the following error when running the application:
{ Error: socket hang up
at TLSSocket.onHangUp (_tls_wrap.js:1137:19)
at Object.onceWrapper (events.js:313:30)
at emitNone (events.js:111:20)
at TLSSocket.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1064:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
code: 'ECONNRESET',
path: null,
host: 'www.politi.dk',
port: 443,
localAddress: undefined }
However if I change the URL to any other URL it works and I get the following:
statusCode: 200
statusMessage: OK
I have tried passing other URL's on the politi.dk domain, which doesn't work either. Therefore I can conclude that there's a problem with this domain, when requesting pages via the Request module. The strange thing is just, that it worked up until recently. What can cause this problem? Can some changes in settings be made to the server of politi.dk, that is causing this now? I find it hard to find anything helpful on Google. I found the nodejs-what-does-socket-hang-up-actually-mean thread here on SO, which is the exact same problem. But the answers doesn't help me much.
Anyone?

Node.js proxy server socket hang up

I am using Node.js as a proxy server and I cannot get rid of the following error. Can anyone familiar with NodeJS assist in finding a solution. Each time this happens I have to restart the .js proxy.
events.js:71
throw arguments[1]; // Unhandled 'error' event
^
Error: socket hang up
at createHangUpError (http.js:1264:15)
at CleartextStream.socketCloseListener (http.js:1315:23)
at CleartextStream.EventEmitter.emit (events.js:93:17)
at SecurePair.destroy (tls.js:938:22)
at process.startup.processNextTick.process._tickCallback (node.js:244:9)
Thanks in advance for any assistance.
This error happens sometimes, it's normal, because client/server can break connection by himself in a wrong way. You could listen for 'error' event on socket, so you can catch error and don't restart whole process.
So in my case the socket hangup was with the post request and the get request worked fine from post man and as well as from browser
The post request body which were sent from 1st server was the raw JSON, so the proxied server will not be able to parse the body.
We need to stringify the body on request, below code will help us to stringify and send the data to proxy server on trigger of request
const proxy = httpProxy.createProxyServer();
proxy.on('proxyReq', proxyRequestHandeler);
proxyRequestHandeler = (proxyReq, req) => {
if(req.body) {
const bodyData = JSON.stringify(req.body);
// In case if content-type is application/x-www-form-urlencoded -> we need to change to application/json
proxyReq.setHeader('Content-Type','application/json');
proxyReq.setHeader('Content-Length', Buffer.byteLength(bodyData));
// Stream the content
proxyReq.write(bodyData);
}
}

How can I get Nodejs twitter-passport behind proxy?

I'm testing twitter auth using passport-twitter's nodejs module. I already created my Twitter app and configured everything like especified by tutorials. The problem is that i'm behind a corporative proxy and as far as i know, node doesn't have any global proxy configuration and do not respect system proxy configuration. This is the output i get when i try to authenticate using twitter-passport:
InternalOAuthError: Failed to obtain request token
at Strategy.OAuthStrategy._createOAuthError (/home/droid/WebstormProjects/passport-social-master/node_modules/passport-oauth1/lib/strategy.js:390:13)
at /home/droid/WebstormProjects/passport-social-master/node_modules/passport-oauth1/lib/strategy.js:244:40
at /home/droid/WebstormProjects/passport-social-master/node_modules/oauth/lib/oauth.js:543:17
at ClientRequest. (/home/droid/WebstormProjects/passport-social-master/node_modules/oauth/lib/oauth.js:421:9)
at emitOne (events.js:77:13)
at ClientRequest.emit (events.js:169:7)
at TLSSocket.socketErrorListener (_http_client.js:265:9)
at emitOne (events.js:77:13)
at TLSSocket.emit (events.js:169:7)
at connectErrorNT (net.js:996:8)
at doNTCallback2 (node.js:452:9)
at process._tickCallback (node.js:366:17)
So far, i have tried to set a global tunnel using this:
var globalTunnel = require('global-tunnel');
globalTunnel.initialize({
host: 'proxy.example.com',
port: 8080
});
In this case, all I get is this message:
TypeError: Request path contains unescaped characters.
at new ClientRequest (_http_client.js:54:11)
at exports.request (http.js:31:10)
at TunnelingAgent.http.request (/home/droid/WebstormProjects/passport-social-master/http-proxy.js:36:15)
at TunnelingAgent.createSocket (/home/droid/WebstormProjects/passport-social-master/node_modules/tunnel/lib/tunnel.js:116:25)
at TunnelingAgent.createSecureSocket [as createSocket] (/home/droid/WebstormProjects/passport-social-master/node_modules/tunnel/lib/tunnel.js:188:41)
at TunnelingAgent.addRequest (/home/droid/WebstormProjects/passport-social-master/node_modules/tunnel/lib/tunnel.js:80:8)
at new ClientRequest (_http_client.js:139:16)
at exports.request (http.js:31:10)
at Object.http.request (/home/droid/WebstormProjects/passport-social-master/http-proxy.js:36:15)
at Object.globalTunnel._defaultedAgentRequest (/home/droid/WebstormProjects/passport-social-master/node_modules/global-tunnel/index.js:211:38)
at Object.exports.request (https.js:173:15)
at Object.globalTunnel._defaultedAgentRequest (/home/droid/WebstormProjects/passport-social-master/node_modules/global-tunnel/index.js:211:38)
at exports.OAuth._createClient (/home/droid/WebstormProjects/passport-social-master/node_modules/oauth/lib/oauth.js:256:20)
at exports.OAuth._performSecureRequest (/home/droid/WebstormProjects/passport-social-master/node_modules/oauth/lib/oauth.js:371:19)
at exports.OAuth.getOAuthRequestToken (/home/droid/WebstormProjects/passport-social-master/node_modules/oauth/lib/oauth.js:542:8)
at Strategy.OAuthStrategy.authenticate (/home/droid/WebstormProjects/passport-social-master/node_modules/passport-oauth1/lib/strategy.js:242:21)
I tried too the semi-global proxy(http://blog.shaunxu.me/archive/2013/09/05/semi-global-proxy-setting-for-node.js.aspx) by Shaun Xu. I imported the modified require.js everywhere but nothing is working. So i'm getting without options...
Your problem :
An old dependence in global-tunnel. see global-tunnel Issue #13
Quick solution : updating node-tunnel dependency to 0.0.4 does not seems to work at the end since using it will redirect all requests to internet proxy vs only authentication requests.
It's an 'old' post but if someone has a similar problem:
Use case :
Server is behind an internet proxy
Server access local resources via http/https, for example an elasticsearch server that must not be requested via proxy
You are using passportJs with Authentication module witch requires an external token validation, for exemple google on https://accounts.google.com
Problem :
By default NodeJs does not take into account any system proxy setting
Company proxy does not forward internal request via internet proxy (eg: if proxy is set, all requests are going to internet proxy, not anymore in intranet or localhost)
Solution not working :
Setting system env HTTP_PROXY or HTTPS_PROXY does not works for 2 reasons:
NodeJs ignore them
Proxy is set for all requests, and you want only one specific request to go through proxy
Using a global proxy (global-tunnel or similar) will fails because they redirect all requests adding their own Agent (request.options.agent) and thus will forward all your traffic to your proxy. Not desired.
A Solution :
Use a similar code as global-tunnel, adding a whitelist to proxy only some hostnames.
I ve implemented and tested this solution with success : see https://github.com/bloublou2014/httpx-proxy-agent-config
Usage example :
var proxy = require('httpx-proxy-agent-config');
proxy.install({
http_proxy: 'http://yourProxy:3128',
https_proxy: 'http://yourHttpsProxy:3218',
// example for passportjs Google OAuth2 + Google+
whitelist: ['www.google.com','accounts.google.com', '173.194.66.95', '74.125.192.95', '209.85.201.95', 'www.googleapis.com']
});
// try to access a page via http request :
var http = require('http');
http.get('http://www.google.com', function (response) {
var body = '';
response.on('data', function (d) {
body += d;
});
response.on('end', function () {
console.log("Body=", body);
});
});
// try to access a page via https request
var https = require('https');
https.get('https://www.google.com', function (response) {
var body = '';
response.on('data', function (d) {
body += d;
});
response.on('end', function () {
console.log("Body=", body);
});
});

How to get twitter api access using http request from browser

Is there any way to get the data from twitter api for any location?
var sanFrancisco = ['79.86','12.62','80.28','13.21']
var stream = T.stream('statuses/filter', { locations: sanFrancisco })
stream.on('tweet', function (tweet) {
console.log(tweet)
It gives an error:
Error: Bad Twitter streaming request: 401
at Object.exports.makeTwitError (/home/file_upload/node_modules/twit/lib/helpers.js:74:13)
at IncomingMessage. (/home/file_upload/node_modules/twit/lib/streaming-api-connection.js:95:29)
at IncomingMessage.emit (events.js:117:20)
at _stream_readable.js:943:16
at process._tickCallback (node.js:419:13)
I assume you followed the usage guide here https://github.com/ttezel/twit#usage and created the twit object with suitable OAuth keys?
Another possibility for this error is if your clock is out of sync, the twitter endpoint will reject requests if the time is out by a few minutes. Check your computers time is correct.

Resources